The REF-erendum of 2018: building the case for remain*

Imagine that academics across the United Kingdom were granted a referendum on membership of the Research Excellence Framework. How would they vote?

This is a facetious question, since there’s never been anything democratic about the way the government distributes research  funds. But if the mood of academics swings against the REF, there will always be politicians and bureaucrats prepared to suggest enticingly simpler systems. So a groundswell of opposition, whipped into life by a distaste for all forms of performance monitoring, should give cause for concern among remainers.

A month ago, I would have predicted a strong vote for remain; after recent experiences with the leave movement on social media, I’m not so sure. It’s therefore worth considering what could be done to secure the vote.

 

Project fear

Any good referendum campaign – I think I’ve got this right – needs fear. If there was no REF, that would surely mean the end of Quality-Related funding, currently one part of the UK’s dual-support structure for the distribution of public research funds. That would mean either a loss to the sector of approximately £2 billion per year, a transfer of that £2 billion into grants, or a combination of these options. (Anyone hoping, by the way, that the government might just hand over QR regardless is misreading the definitions of ‘quality’ and ‘related’.)

Regardless, a REF-less world would mean less autonomy for universities in decisions over research development, at a time when the government is already being more intrusive, most notably through the Industrial Strategy Challenge Fund. It would also be disastrous for humanities and social sciences disciplines, which do disproportionately well out of QR and disproportionately poorly out of grants. Toss in the possibility of variable fees by discipline, and HASS departments could face a generation in the freezer. Armageddon my point across?

But maybe we need a more positive message as well. It’s striking just how many academics don’t see any benefits from QR. Some associate it merely with the few hundred pounds of research allowance that their departments hand them. Others resent central decisions to devote QR to grand projects that seem remote and unimportant.

There are lessons to be learned here for VCs and PVCs. Granted, from their perspective all income falls into one big pot; researchers, however, want to feel the benefit of their labours. And it shouldn’t be difficult: for a start, how about universities transparently cost their research leave policies (for those that have them), and connect the dots with QR?

And there may also be some lessons for UK Research and Innovation. RAEs and REF of the past have generated powerful, inspirational narratives of research success for the benefit of politicians and policy-makers. Might some of this effort be redirected towards researchers themselves, who more commonly experience the REF as an oppressive process than a source of inspiration? Perhaps their good will has been taken for granted.

 

#loveREF

On the evidence of recent social media exchanges, some of the most vocal supporters of REF are immigrants. If the UK’s REF-shaped clarity of expectations has offered an international researcher a career-lifeline, to move from a country in which appointments and promotions are more sluggish and opaque, this can engender a warm glow of affection.

By comparison, academics who have known nothing but the REF are less likely to appreciate what it has done for them – creating jobs, expediting promotions, and so forth – and more likely to view those distant pre-REF days with a sense of wonder. Back then, people spent longer on their big ideas, didn’t they? Britain produced big, serious books, and Nobel prize winners. Academics were left alone to just get on with it.

Too much detail in this table, but it demonstrates the rapid growth of UK research impact, by citation, in the RAE-era. From Adams & Gurney, Funding selectivity, concentration and excellence.

One way of confronting this nostalgia is through data. There is evidence that British research in general was lagging through the 1980s. The surge that has brought the UK to its current position can be traced back to the effects of early RAEs beginning to bite within universities. The UK now UK ranks first amongst its comparator countries by field-weighted citation impact, and this can be explained in part by its broad research base.

But data alone won’t convince people to love the REF. Leavers have the more compelling stories: of poor management in the name of REF preparation; of ECRs being suspended in casualization as they try to build sets of publications to sell to an employer that will immediately push them to repeat the performance for the next cycle.

So there’s work to be done for the remain campaign. Pointing to Professor Ego who has just secured a big pre-REF pay-rise won’t necessarily seize the imagination of a lecturer losing money on the picket-line. UKRI might do well to take some initiative, aiming to make the REF less forbidding. A training programme for line-managers might help, or some research into the effects of REF on appointment decisions. REF might also align itself more explicitly with ethical frameworks such as Athena SWAN or the Vitae Concordat. The ‘real-time review’, announced this week, looks like it might be a step in the right direction.

REF managers may feel burned by the reception of their well-meant proposals on portability, intended to iron out distorting effects of their system on the job market. But now that the rules for REF 2021 are settled there is space for further efforts to inform academic culture. Maybe love is unrealistic; trust, actually, would probably suffice.

 

Close your eyes and it might all go away

There are more important debates to be having than the one about REF. We should be focusing on the overall quantity of public research funding. Then we might consider the balance between grants and QR. And then again there’s the desperate uncertainty over European Union collaboration. Fussing about the mechanism for distributing QR feels insular, and risks giving the impression that academics are not much bothered with accountability.

But perhaps the REF-erendum is a process we need to embrace all the same. The remain camp can win, yet they may need to engage more openly with their critics’ concerns. These won’t just disappear; and if the remainers lose, nobody can expect a second vote on the withdrawal terms.

Advertisements

Fixing Stern

Now that we’ve had a few weeks to get used to the Stern recommendations for the next Research Excellence Framework, some of its underlying principles look immovable. I considered these in my last piece on Stern; it looks like we’re heading for an ‘everyone-in’ REF, and portability looks dead. I expect that too many powerful interests will be supporting those, and other, recommendations.

But I also think there are flaws that need fixing – and they can be fixed.

 

The Early-Career Researcher dilemma

I work at a university that makes temporary appointments every year. We make permanent appointments whenever we have recurrent income to support them, but temporary contracts remain a basic fact of life. Many ECRs will experience several years of temporary contracts – perhaps interspersed with unemployment – before achieving permanency. Stern’s recommendations, as Kirsty Rolfe has eloquently outlined, are hugely destabilizing to people currently in this position.

Indeed following Stern’s recommendation to the letter, a publication by an ECR will be returnable by the university s/he was at when it was contracted. As a result, I could be on the phone in 2019, tracking down all the bright young things we have employed over the years, asking them very politely for copies of their book contracts.

Will that be embarrassing? No, it will be way beyond that. How could we argue that our level of ‘investment’ in these careers warrants such a dividend? And of course this embarrassment multiplies in the case of teaching-only appointments. Stern’s report barely grasps the fact that these exist as a starting-level job for many ECRs.

And let’s not forget that some of these people may not have landed the permanent jobs they deserve in the interim – in part because they will have blown their best post-PhD shots on a temporary employer like mine. Yeh, those people will be happy to hear from me.

But surely there’s a simple answer; indeed it seems so obvious I can’t really believe it needs saying. Any publication contracted while the author is on a temporary contract should remain portable, while temporary people in post at the census date may be submitted as in the past. There are universities that might lose a little from that – notably those that fund a lot of junior research fellowships – but I can’t see how anyone could dispute the ethics of it.

 

‘The dog ate my book contract’: or a note on portability

The recommendation against portability of outputs changes the dynamics of the job-market; I’ve commented on this before. But even if we accept this principle, one practicality that still looks in need of a fix is the recommendation that a publication ‘belongs’ to the university at which the researcher was based when the contract was signed. The principle here is that the REF should recognize the investment made by universities; however, in practice we could get some silly and counter-productive results.

Let’s just say (and this is a common enough scenario in the humanities) that I get offered a publishing contract in the next few weeks for a monograph that may not be finished until 2020. And let’s also say I’m also looking for a new job. Who’s going to pay me to finish a project for my old employer?

As a result, I expect we might find rather a lot of contracts being eaten by dogs. ‘Please, Oxford University Press, can you send me a new one, maybe dated 2017?’ Or: ‘how about we change the terms a bit – even cut my royalties – and call it a freshly negotiated contract?’

There’s a solution. We could retain the principle of Stern, but use publication dates rather than dates of contracts. That’s still a profound change from the present system, but seems to me slightly more flexible and infinitely more feasible. I also rather suspect it will happen. Stern’s recommendation has the whiff of an ambit-claim: like, give the critics some ground on the method, while the principle glides through unscathed.

 

A comment on rent-seeking

rent-seeking   n. Econ. the fact or process of seeking to gain larger profits by manipulating public policy or economic conditions, esp. by means of securing beneficial subsidies or tariffs, making a product artificially scarce (OED)

There’s a streak of moralism running through the Stern report, directed against high-achieving academics who choose to move between institutions or seek pay-increases on the eve of a REF. This, Stern says, is ‘rent-seeking behaviour’.

All I can say is that some of these things are not as simple as they might look. I came from Australia to Leeds in 1999 (on the eve of RAE 2000) for a three-year job, then moved almost immediately for a better job at Exeter. Was that ‘rent-seeking behaviour’? What it felt like was being an ECR looking for a position that gave me job-security and a salary commensurate with my research record. I expect that a lot of cases of ‘rent-seeking’, ‘poaching, and what have you – all terms coined by disgruntled employers – might similarly be positioned as quite reasonable acts of career-advancement.

So Stern ditches meritocracy for austerity-speak labour-market constriction. Actually, I think Stern will win on this one, and we will have as a result a less open and flexible university system, which also in due course becomes less competitive internationally. (See Timothy Devinney’s excellent piece on this.) So on this point I don’t see an easy fix; however, I’d certainly suggest it would be worth thinking about ways of moderating the message. Across the country, and across the world, researchers are listening to this stuff.

Research, researchers and the job market: thoughts on Stern

Plenty of positive things have already been said about Nicholas Stern’s review of the REF. In a summer of uncertainty, Stern has provided continuity, notably in his commitment Sterntp3to peer review and the fundamental value of publications. If the report’s recommendations are accepted, the REF will remain an exercise broadly underpinned by academic perceptions of research quality.

But there are also recommendations that will alter relationships between research, individuals and departments. They could also have a significant impact on the job market, particularly in humanities subjects. The impact won’t necessarily be bad; indeed Stern clearly intends to fix some perceived problems. Yet it’s worth thinking through precisely how they would affect us.

 

We’re all in it together

Recommendation 1: All research active staff should be returned in the REF

The principle of universal inclusion is powerful; I like it. But Stern’s intention, here and elsewhere, is to privilege the department as a unit of research activity over the individual as a sole producer. Hence his further proposal that, while all research-active staff are submitted, some people may submit many items (maybe up to six) and others no items at all. The average (probably two items per researcher) is what will matter.

How would these proposals affect us? At some places there will be increased pressure to push people onto non-research contracts. That’s not necessarily the case, since we will still have one eye on the multiplier effect for QR funding calculations; however, for departments just seeking some respectability in league tables while focusing largely on teaching, it will probably happen.

Meanwhile, decisions about workloads and distribution of resources (including research leave) will be interesting. Ditto advice – and contractual requirements, for some of us – concerning the quality and quantity of outputs. For years I’ve argued that the only thing that really matters, when hiring or promoting, is a researcher’s capacity to produce four-star work. Maybe now I win that argument.

I also think there’s reason to think very carefully about the report’s proposal to abolish ‘special circumstances’. Stern suggests that these will come out in the wash once we shift the focus from individuals to departments; departments will adjust expectations across a group and muddle through. This is to think very clearly from the perspective of an administrative system, but not at all from the perspective of an individual researcher. For many people – and more women than men – the system of special circumstances has been perceived as clear and supportive. Sweeping special circumstances under the carpet cannot be equivalent to a credible equality and diversity policy.

 

You can’t take those outputs with you

Recommendation 3: Outputs should not be portable

A year ago, I lamented that we can’t take impact case-studies with us when we move jobs; now Stern is proposing that we won’t take publications either. In fact he’s taken a step further: the report proposes an output should belong, for REF purposes, to the university where the researcher was employed when s/he signed a contract for its publication.

The principle here is crucial. Thinking (as Stern consistently does) from the perspective of the institution, research outputs represent returns on investments. It is therefore unjust if these outputs occasionally get carried off to another university in the months before a REF deadline. Moreover, as we all know, such moves can distort both the job market and internal pay differentials.

Well, maybe, but this is to set aside the powerful connection that researchers feel with their publications. It is also to remove one of the key sources of power that academics currently have in negotiations over pay. Whether we like it or not, this recommendation would substantially alter that balance of power, and rewrite equations of ‘market-worth’. I’ll also be curious to see, if this proposal is accepted, whether this shift has an effect on motivation, and hence academics’ famed willingness to work unpaid overtime.

The details are equally important. The focus on the date a contract is signed looks to me like Stern’s committee didn’t talk much to researchers in the humanities. I think I’m right in saying that the period between the signing of a contract and the date of publication is much shorter in the sciences than the humanities. Even humanities journal articles can sit in queues for two or three years. And the report seems entirely unaware that some of the most REF-valuable humanities publications – monographs or (in my discipline) critical editions – may be contracted years before they are completed, let alone published.

So consider the effect on the job market. That old interview chestnut, ‘Is your monograph contracted?’, takes on radically new meaning. Somebody who is committed to a pipeline of publications, in a way that last week looked sensibly professional, may now look much less attractive. Can it really be right that I could spend my first three years at a new university being paid to finish work for my old employer? And do we really think this will end ‘game-playing’? Surely we’ll just end up with different kinds of games.

And then there are the blindingly obvious problems presented by early-career researchers. Who owns that first monograph, that might currently win someone a first permanent job? The current REF rules actually work fine for productive ECRs: it makes sense for universities to employ them. This aspect of the report looks very odd, and really should be fixable.

 

And big is better

These proposals are better news for bigger departments. For a big department that has always aimed for 100% submission anyway, these changes will be easy enough to manage. My immediate response to reading the report was to take an evening off. But if you’re in a smaller department, perhaps with a low submission rate in 2014, perhaps with a number of people who might reasonably claim ‘special circumstances’, the next REF will today look significantly more challenging.

Four-star impact

I’ve spent some time recently reading four-star humanities impact case-studies from REF 2014. This is possible because some departments achieved a perfect score for impact, so we know that all their case-studies achieved the highest grade.

As we turn our thoughts to 2021 (or thereabouts), and as impact becomes accepted as part of research assessment beyond the UK, it’s worth pausing to ask what lessons can be drawn from this evidence. My analysis is subjective and impressionistic: really just picking out some patterns that struck me, in the light of discussions I’ve had over the years with colleagues. The departments I’m considering are: English at Bedfordshire, Newcastle, Kingston, Swansea; History at Hertfordshire; Modern Languages at Swansea.

 

Individuals or departments?

One hypothesis with which I began was: if we’re trying to succeed with impact, it makes more sense to think about four-star departments than four-star case-studies. In other words, I suspected that departmental cultures were more important than star individuals.

The evidence is equivocal. At some places, it appears that impact is delivered brilliantly by a minority of staff, albeit with appropriate institutional support. That’s probably the way most of us are working, in fact, not least because of the time that impact-oriented work absorbs. But some of the ‘impact templates’, outlining methodical efforts and commitment across a department, are instructive. For evidence of how to get it right, I’d recommend History at Hertfordshire.

 

Impact and engagement

One of the great academic corridor put-down lines these days is: ‘Oh, that’s not impact; it’s just public engagement’. Seriously, you wonder what planet we’re on at times (and note, by the way, an excellent argument against allowing the assessment cart to be put before the impact horse in the THE). Nonetheless, this line has a point. It doesn’t make sense, from the perspective of time-management, for overworked people to be knocking themselves out on activities that they mistakenly believe will feed into an impact case-study.

So we focus on impact – on having identifiable effects on identifiable groups – and scale back on engagement. Right? Well, the evidence suggests much more porous dividing-lines between impact and engagement than many would like to believe. Take the historian preparing an exhibition on the basis of a chance research discovery. The aim will be to engage as widely as possible, but four-star impact may still be the result.

 

Locality matters

In the early days of impact, one theory was that it would favour London universities. Not so. Some of the most compelling work was achieved by universities a long way from the cultural centre, with powerful regional commitments.

Take English at Newcastle. This department, over many years, identified ways of involving itself in the region. One example of this commitment is its partnership with Seven Stories, the national centre for children’s books: a partnership evident in staffing decisions, grant activity, doctoral studentships, and so forth. Or take English at Swansea, where a longstanding, passionate commitment to Welsh literature and culture shines through in two of their three case-studies.

 

Creativity matters

It’s common sense really: creative people are geared towards engaging with the public. This doesn’t mean that all creative practitioners in universities will be delivering in terms of impact, but it puts them in an excellent position to do so. Take creative writers. Publishing a novel, selling some copies, and doing a few literary festivals may not necessarily amount to much in terms of impact. (Although – and we need to stress this over and over – creative outputs did do rather well in REF 2014.) But if someone writes about a topical or controversial subject – for instance, Jackie Kay (Newcastle) writing about her experiences as a child of Nigerian birth, adopted by white parents in Glasgow – this can lead to important and powerful impact.

 

Questions of longevity

It’s commonly observed that the time-lag between research and impact in the humanities tends to be quicker, on average, than in the sciences. But some of the most impressive case-studies are the product of many years of commitment, and bring benefits that will equally span years. For instance, consider the determination of academics in English at Swansea to bring works of English-language Welsh literature into the public domain, via a series of books that is part-funded by the Welsh government and now sits in every school in Wales. That was an extraordinary achievement.

In this context, of course, it will be fascinating to see how the rules for 2021 deal with the question of ongoing impact from case-studies submitted in 2014. Some of the best impact endures, and hopefully this will be recognized.

 

Questions of cost

Impact costs time and money. In many cases it’s funded by research grants, but across the board it’s apparent that the better departments are underwriting, to a considerable extent, staff time and costs. I’ve commented before about the ‘impact industry’: the advisers, professional case-study writers, and so forth. I think there’s cause to be sceptical about all of that, yet there’s also cause to admire the impressive, moving work documented in these case-studies. The challenge for managers is to find ways of ensuring such work is adequately resourced, at a time when academics and their departments are stretched by competing demands.

 

You can’t take it with you

I’ve noted before the effect upon the impact agenda of the rule, in REF 2014, that if an academic switched universities mid-REF cycle, s/he would take his/her outputs, while the university would retain any impact case-study. That skews the value, to the individual, of impact-related work. Why would a university will spend big money to recruit a major impact-star, as opposed to an output-star, on the eve of  a REF?

One rumour about the next REF is that there may be a perverse solution: perhaps outputs, as well as impact, may be retained for submission by previous employers. Well, let’s wait and see. Such a move might well help keep a lid on wage inflation at the top end; although the implications for early-career appointments would need consideration.

The paradoxes of neo-liberalism in UK higher education

The latest argument that we should be worried about a crisis in the university system comes, via The Guardian, from the US consultant Karen Kelsky. Her key concerns are student debt and casualization of the academic labour-market. The UK, she claims, is barrelling down a road familiar from the US.

Student debt? Well, yes: absolutely, and let’s not forget it. But I want to reflect here on the arguments about casualization and the fears for early-career academics, because it seems to me there are some curious paradoxes here. For Kelsky, the ‘neo-liberal’ structures of UK higher education, such as the Research Excellence Framework, are all part of the problem. But as I see things, they are also having contrary effects, helping to hold at bay some of the economic logic that has driven things in the US.

So let’s consider some effects of two of those neo-liberal monitoring structures, both unique to the UK: the REF and the National Student Survey.

 

The REF and the academic job market

Casualization of academic labour makes good economic sense. Why pay someone a full-time professional salary when you can hire in perfectly well qualified temporary lecturers and pay them only for the teaching they do? That logic has taken root in many US universities; it will most likely drive the growth of private universities in the UK. So there definitely are reasons to be concerned.

But the REF posits a contrary logic, along the lines: why appoint temporary and part-time teaching staff, when you could appoint someone who will contribute to the REF? And that appointment – not always, but more often than not – will be permanent. Certainly that’s my experience. We always have temporary lecturers – to cover for people on funded leave, or maternity leave, and so forth – but we appoint to permanent, research-active posts whenever we can.

And it seems to me that the REF is also a friend to early-career academics. There’s a strong, REF-guided logic to appoint younger people, publishing high-quality work often straight from their doctoral research. It’s not about quantity: four decent pieces in 6-7 years is not unreasonable, and ECRs will typically require fewer than four. The principle of peer-review, meanwhile, remains strong, underpinning the commitment to rewarding quality.

And this all means that universities will generally gain more benefit, for relatively low cost, appointing junior lecturers, as opposed to appointing senior people. This does stimulate the job-market, albeit in an uneven pattern: better in pre-REF years than others. It won’t create jobs for everyone finishing PhDs (and there is a genuine debate to be had over whether we are educating too many smart young people to doctoral level), but it surely works against the logic of casualization.

Kelsky argues also that the REF drives us all towards performance targets, and leads to the persecution of great minds who work slowly. Yes, that’s an old and not invalid argument. But here’s the paradox: if the goal is to create a structure that is more open to early-career academics there is actually a value in ensuring that those in mid-career and late-career are actually doing all parts of their job. However much some of us might resent it, the REF helps with that.

 

The National Student Survey

The National Union of Students has proposed a boycott of the NSS. Actually there’s a logic to this: they argue that if the NSS is to be linked into the Teaching Excellence Framework, and if the latter is to be used to determine differential fees, then the Survey’s original purpose of feeding back to universities on their performance will in practice be superseded by its use as a vehicle of marketization. As someone who has seen the NSS improve the quality of education over many years, I fear a boycott would be self-defeating. But I can see the point.

Yet the NSS – and in due course the TEF, presumably – also works contrary to the forces of casualization that Kelsky bemoans. The NSS gives students some power, and in my experience students tend to be fairly clear about their desire to be taught by fully-qualified and fairly employed lecturers. That’s not to say that we haven’t had superb feedback in my department, year after year, on our (very well trained) graduate teaching assistants. But it is definitely one reason why we employ very few people who look like US-style ‘adjuncts’: who are, typically, people with PhDs, paid to drop into a campus to deliver particular courses, and often working simultaneously at multiple universities.

Do we have people in this category in the UK? Absolutely we do; and I agree that it’s a problem. But I’m yet to see hard evidence that it’s getting worse. And, as I see things at least, another of those neo-liberal monitoring devices, the NSS, is working to some extent counter to the logic of casualization.

 

I’ve written before about the temptation to draw easy parallels between the US and the UK (in the context of the so-called ‘crisis of the humanities’). I think there are genuine problems in the US, which affect all of us one way or another; and I think there is always cause to be vigilant about developments in the UK system. But there are paradoxes in some of these arguments, and structural forces pulling against what may seem like an incontrovertible economic logic. As much as the UK’s various monitoring systems may at times feel oppressive or frustrating, it just might be the case that they have some positive effects on the sector.

 

REF-cycling

One of the characteristics of the REF/RAE is that we always spend two or three years playing a game with no goalposts. It is now over two years since the deadline for REF 2014 submissions. It will be close to three years by the time we know what form REF 2020-something will take.

This hiatus poses a question: to what extent should we plan when we don’t know the rules? Responses might be positioned on a spectrum. I’ve heard of departments where academics are told to keep calm and carry on doing good research. There’s a simple decency to that. And there are others where plans, drafts and publications are being reviewed, monitored and graded, more assiduously than ever.

My university is somewhere towards the latter end of the spectrum. That’s just the way we are. And I feel myself, at the level of a department, that it would be irresponsible not to ensure that colleagues are carefully assessing the possible consequences of decisions about research plans. Research time is precious – and REF contributions, let’s face it, have a huge bearing on career progression in the UK – so these decisions will matter.

That’s why we’ve been spending some time recently, in my department, listening to REF 2014 panellists and thinking about our planning. I’ll outline here – in a post that is admittedly humanities-heavy and UK-centric – some of what’s been on our minds.

 

Originality, significance, rigour

The REF criteria are simple enough, attuned with what most of us value about academic work. (For some helpful expansion, see the Panel D criteria, p. 88.) Originality, significance, originality: and ‘significance’, it’s worth noting, should not be confused with ‘influential’, since a publication may be significant, in the eyes of a REF panel, even in a small area and with a small readership. (That’s one way metrics could change things.)

In my department, though, one of the lessons we’ve been learning is that we’ve relied too much in the past on proxies of quality. Most of all, we’ve been overly swayed by length. In our REF 2014 planning, we made easy assumptions that monographs would be graded four-star. Now we’re accepting that there were plenty of three-star monographs submitted to the English panel in 2014, and even some two-star ones. Meanwhile, there were plenty of articles graded at four-star: something we were equally reluctant to predict. (Equally, plenty of editions, collections of essays, creative works: there are some myths about what ‘matters’ in the REF that are well worth exploding.)

What lessons might we draw from this? Firstly, it seems to me there are risks of over-reacting, or of seeing the positive message (‘my articles could be four-star’) and missing the negatives (‘my monographs might not be’). Secondly, there are some really interesting mentoring challenges, because if I have colleagues devoting a lot of time to writing monographs, I want to do all I can to help them ensure those monographs are as good as they can be. The alternative might be deciding, in a future mock-REF process, that some monographs don’t cut it. And that would be a tough, potentially divisive message.

Thirdly, it perhaps changes the mind-set, quite common in the humanities, that goes along the lines: ‘focus on the monograph and sort out some other items along the way’. If an exceptional article may rate better than an unexceptional monograph, we need to be thinking very carefully about all our items.

Fourthly – and this isn’t strictly relevant, but I like to say it whenever I can – we need to think about what not to do. There’s no point showing up for a REF with a pile of two-star publications and a case of work-related stress.

 

Game-playing or dodgy rules?

There was a lot of talk about ‘game-playing’ after REF 2014, much of which was sour-grapes silliness. But it’s becoming pretty clear that some of the rules created the effect of game-playing, regardless of the players’ intentions.

Exhibit A is ‘double-weighting’. Asking units of assessment to decide whether or not to request double-weighting for an item distorted the playing field, hurting those who interpreted the requirements more conservatively than others. The solution next time must be to give panels more authority to determine what is in the best interests of the submission.

Then there’s ‘repetition’: the submission of work that had previously been published in another form. We were appalled to be criticized for this, since we had been very careful; however, it now seems rather a lot of English departments received the same feedback. The application of the rules, let’s say, had a punctilious vigour. I’d hope we can do better next time – but in the meantime, it pays to know.

 

The impact industry

Allow me to quote again my favourite sentence from the Green Paper: ‘We must also address the “industries” that some institutions create around the REF and the people who promote and encourage these behaviours.’ And hence to impact.

The impact industry is building. There are impact gurus who will help us maximize our impact, case-study writers to support us as we prepare for a REF, and training programmes to help us understand it all. My impression is that the level of expertise around impact in the humanities is rather less reliable than it is for the sciences and social sciences. Although I’ve made this point before, it therefore remains crucial to nurture and respect experience – knowledge of what works, and what was recognized in 2014 – within our disciplinary communities.

Thinking ethically, one wonders whether all the money devoted to the impact industry might be better spent. But thinking rationally, this was surely bound to happen. I mean, what did they expect? There was indeed a touch of utopianism to that Green Paper sentence.

 

Might the rules all change? I wouldn’t bet on it. I’m predicting a REF 2021 that will look rather like REF 2014: with maybe a few of the creases ironed out, and probably a few more ironed in. But I might be wrong.

Paul Nurse’s Utopianism

Paul Nurse’s report on the research councils, Ensuring a Successful UK Research Endeavour, has left some readers bemused. It doesn’t adhere to the conventions of its apparent genre: no executive summary, no tables, very few references, and a personal voice guiding the reader throughout. As I read it – in my first ever blog-post with an embedded play-list – Nurse’s report is a form of utopian discourse.

 

Research UK: total control over you

Most official reports need at least a modicum of utopianism, to the extent that they are required to identify problems and seek solutions. But few betray the burning idealism of Nurse. ‘Science,’ he writes, ‘is a high calling in the pursuit of truth that needs to be pursued in a proper and ethical manner.’ Looking back to the ‘new science’ of the seventeenth century, he notes that ‘Francis Bacon argued that science improved learning and knowledge which “leads to the relief of man’s estate”’. Today, it promises solutions to our ‘grand challenges’.

The report is also profoundly idealistic in its assessment of how systems of research funding might be made to work. Nurse dreams of a world in which there is ‘effective dialogue’ between scientists, politicians and public. He dreams of a world in which universities exercise appropriate quality-control over grant applications, so that plunging success-rates do not render existing funding mechanisms untenable. He dreams of a world in which the best researchers provide intelligent peer reviews of grant applications, and funding panels operate consistently and provide ‘constructive feedback’ to applicants. He dreams of a world in which monitoring systems – even, yes, researchfish – work smoothly. He even dreams of a world in which the discourse of researchers is governed by ‘courtesy’. That’s right, courtesy.

These are noble goals. And yet, as Bacon’s contemporary William Shakespeare appreciated, there is another side to utopian discourse. In The Tempest, the dozy courtier Gonzalo displays his own courtesy by rehearsing a vision of a perfect state. His utopia will be devoid of property, hierarchy and laws, yet is predicated on his own status of ‘king’. As one of his companions notes, ‘The latter end of his commonwealth forgets the beginning’.

The realization of Nurse’s vision may not collapse so utterly on a paradox, but it depends similarly on a benign magus: ‘a highly distinguished scientist, capable of delivering a managerially efficient organisation and of interacting effectively with Government’. This person will be head of ‘Research UK’: a new body built upon the existing structures of RCUK, but with more substance and resources. S/he will be the ‘Accounting Officer collectively for all the research councils’. S/he will also have a budget: funds ‘needed to respond effectively to epidemics, volcanoes, flooding and earthquakes for example’.

So maybe that’s more Prospero than Gonzalo, watching over us with his impeccable map of research activity, and scanning the horizon for natural disasters. Nothing to worry about then, is there, Shakespeareans?

 

The truth about scientists

One question bound to be asked by those of us in the arts and humanities, or even the social sciences, goes along the lines: when is a scientist a scientist?

Nurse opens his report with a definition: ‘In this review the terms “research” and “science” are usually used in the context of the entire academic landscape, reflecting the Latin root, “scientia”, meaning knowledge’. That’s appropriately Baconian, taking us back to a time when the quest for knowledge barely recognized disciplinary borders. More pertinently, perhaps, it is aligned with the discourse of the national ‘science’ budget.

But what about that qualifier, ‘usually’? When might Nurse be using ‘science’ instead in its more common modern sense? When might his ‘scientist’, as a seeker of knowledge, actually be a scientist, as someone operating predominantly in the STEM disciplines?

I never have much time for ‘crisis of the humanities’ conspiracy theories. It’s also fair to say that as long as the AHRC accounts for (on my reading of 2015-16 figures) just 3.6% of the total research councils’ budget, we’re barely worth the fuss and bother of a robbery. But there is just enough ambiguity in the Nurse report to unnerve us. I’d bet the head of RUK will be, well, a scientist. And I’d expect the RUK budget – ditto the (related?) ‘Global Challenges Research Fund’ – to be weighted heavily towards the, let’s say, sciences. This is not the disaster for non-STEM that it might have been, but any humanities researcher will be bound to point out that signs – like words, usually – matter.

 

A Little Green

The Green Paper also had at least one utopian moment. It said, in its slim chapter on research funding: ‘Our ambition is to reduce bureaucracy and release our scientific and research leaders from administrative burdens as far as possible.’ More specifically, it targeted REF preparations: ‘We must also address the “industries” that some institutions create around the REF and the people who promote and encourage these behaviours.’

Absolutely: far too much resource is being devoted to the management of research (and impact), which could better be spent on the thing itself. But I have two questions. Firstly, how do they think they’re going to stop us? Unless (and here’s an interesting thought) someone finds a way of altering the system so as to negate such activities, they will surely remain responsible management by another name.

Secondly, can Nurse help with this agenda? As I read his report, the thought has barely entered his mind. Indeed his desire for a thorough ‘mapping [of] the UK research landscape’, and commitment to fixing the imperfections of grant-awarding systems suggests the contrary. Bureaucracy, for Nurse, is not a dirty word.

 

There’s a lot to like about Paul Nurse’s idealism and advocacy of researchers. Maybe this is one reason why commentary on the report has been slight: there’s not much to get grumpy about. But utopianism is an unreliable foundation for policy. In all of the 499 years since Thomas More published Utopia, I’m not aware of a single example of utopian discourse producing a utopian state.

The genius of the Green Paper

Fulfilling Our Potential: Teaching Excellence, Social Mobility and Student Choice, the government’s Green Paper on higher education, will attract a lot of comment. That’s part of the point of a green paper, after all. Yet, for all its flaws, this is not a stupid document; if one peels back the shortcomings, what’s left is a core of canny genius.

This document covers a lot of ground. In the longer term, the chapters about market flexibility may be the most significant. There are also the major structural changes: the end of HEFCE, the birth of the Office of Students, an apparent reprieve for the QAA. There’s an impressive emphasis on widening participation, a frontal assault on the degree classification system, even a section on student finance and sharia law. And the final chapter on reseearch funding is worth a look.

But I’ll leave some of that for another day. In this post I’ll focus solely on the Green Paper’s proposals for a Teaching Excellence Framework.

What was the problem?

The Green Paper’s central premise is that too many universities are prioritizing research at the expense of teaching.

That’s debatable; it’s not what I see in my patch, nor does the Green Paper do much more than assert the position. In fact there’s rather too little evidence and too much righteous indignation. It’s saying to university leaders, in effect: we sorted out the income stream for you, and now you’re spending the money on research, and getting distracted by global league tables. But if we accept the premise – just for a minute, just for the sake of argument –  we can appreciate the problem the Green Paper is addressing.

The challenge, in short, is to engineer a system that will convince university managers to redirect institutional resources from research to teaching, at a time when the state has no more money to throw at the sector.

The pot of reputational gold

The REF earns money for universities, in the form of ‘quality-related’ payments. For the TEF, the Green Paper shamelessly exploits the undone business from the last round of reform. The failure to index £9000 fees to inflation was a weak act of political neglect, and had to be fixed one way or another.

So this proposal effects a fix, with strings attached. The arrangements for next year are transitional – and, given the neglible rate of inflation, barely consequential – so let’s focus on the longer term. As I read the proposals (and cf. here the visualization at wonkhe.com), the system will look something like this.

  • Any university that earns the QAA kitemark – as almost all established universities do – will be permitted to increase its maximum fees, in any year, by a percentage of the rate of inflation. That percentage remains to be determined; I’d bet it will be at least 50 per cent.
  • In addition, once every five years or so each university will have the opportunity to apply for recognition for higher levels of teaching quality, which will earn the right to increase fees up to the full rate of inflation. This will be a tougher, optional test.

So here’s the clever bit. The financial benefit of applying for the higher levels of recogntion will not warrant the considerable expense, in terms of buureacracy and changes in institutional practices. Yet, while there will be a temptation to opt out, settling for the Level 1 kitemark, the Green Paper is knowingly leveraging ‘reputational advantage’ for all it’s worth.

And in this context, how many vice-chancellors will be prepared to opt out? What sign would that send to stakeholders? To prospective students? I’d call this ‘Johnson’s choice’.

The dimensions of quality

What constitutes success? Some of the measures are expected: student satisfaction, graduate employment rates, retention rates. That’s all fine; these measures command a level of confidence across the sector.

But there’s more: rather a lot more. Much remains to be determined, but there’s some interesting discussion in the Green Paper about learning gain, and also a sensible recognition of associated widening participation implications. Students from lower-performing schools will, on average, gain on those from higher-perfoming schools, if A-Level results are set against degree results. That, by the way, is a courageous thing for a Conservative government to admit.

There’s also an acknowledgement that contact hours can be deceptive; group sizes also matter. And there’s a welcome concern with who’s doing the teaching. The Green Paper targets the prevailing disparity of esteem between outstanding researchers and outstanding teachers. And it asserts that lecturers on full-time contracts, properly trained, will be more effective than casuals. Indeed.

Perhaps most intriguingly, I note the recurrence of a couple of my favourite words: ‘engagement’ and ‘enhancement’. It won’t be easy to assess all this stuff, but the intention is welcome.

The nightmare scenario

Could it all go wrong? The White Paper’s gamble, as I see it, is that a critical mass of established universities will seek the higher levels of TEF recognition, and that standards of education will rise as a result.

But if a significant number of powerful universities opt-out – reasoning that concentrating on research income, international reputation and international students makes more sense than scrabbling after higher-level TEF recognition – we could get some perverse outcomes. Let’s say, for example, that the LSE (second from bottom in the 2015 NSS) decides not to bother, while Liverpool Hope (seventh from top) earns the top TEF grade. That would leave the LSE with lower fees than Liverpool Hope, and the sector in danger of collapsing into a credibility gap.

In this context, the Russell Group’s response to the Green Paper must surely carry some weight.

‘Reducing regulation’: really?

There’s been a lot of talk in recent years about universities winning independence from government regulation. The Green Paper, whatever noises it makes to the contrary, calls a halt to all of that. It may not fully succeed in putting students at the heart of the system – the goal it inherits from its forebears – but it definitely puts the state right back into the mix.

You can’t take it with you: impact 2020

Here’s a pre-REF 2020ish scenario. Professor X and Professor Y get job offers elsewhere. Professor X has two four-star publications, Professor Y has prepared two four-star impact case-studies. The university wants to retain them both, but funds are tight and priorities need to be clear. The impact case-studies are worth eight times as much as the publications, but publications travel with their authors whereas impact case-studies (under 2014 rules, at least) are owned by the university. So Professor Y goes; in fact, s/he probably wouldn’t have had a job offer at that time in the first place.

Hence the response of one of my colleagues, reviewing an application I’d prepared for the AHRC follow-on (i.e. impact-related) scheme. ‘This is all lovely,’ he said, ‘but what’s in it for you?’ He’s not an impact Luddite, mind you; his response was that of someone who has been ploughing this furrow longer than I have, and occasionally wondering why.

I don’t want to argue for more money for the impact-active. The real problem, it seems to me, is the way outputs distort the job market, not the way impact fails to do so. But I suspect that a lot of us who are in the business of managing departments are still grappling with the question of how to plan for 2020ish. How do we best recognize quality impact? How do we support and enhance potential case-studies? And how do we document them?

Recognizing the impact stars

To return to the scenario of my opening paragraph: in actual fact now, not 2019, is the time to buy impact. In the humanities, we learned in REF2014, lead times between research and impact tend to be shorter than for the sciences, meaning that it’s perfectly plausible to employ someone in September 2015 and expect a four-star impact case-study from them four years later.

Are any universities doing this? Not many, after all the pre-REF spending, will have sufficient space in their business plans. There have been a couple of ads for chairs in digital humanities – which is interesting in terms of the growth of a field, and perhaps also suggests a perception that this is an impact-rich field – but I haven’t seen any bold moves more specifically in my own discipline. Some of the more interesting decisions may fall at the lower end of the scale. When looking at applicants for lectureships, how important might it be that a candidate has the capacity to add value to a planned case-study? Might this prompt us to think, in the humanities, a little more than we have traditionally done in terms of research teams?

And how do we recognize those already delivering impact? One question of principle, which has a bearing on workloads and recognition, is whether impact is simply one aspect of research, or something else again. Hence, when allocating workloads it’s not clear whether we expect someone to deliver impact as part of their allotted research time, or whether we need to create more space for the impact-active. We’re still fudging this in my department. For this year I was granted, to support my impact work, ten hours in our 1650-hour annual workload model. That’s a figure so ‘neither here nor there’ that it risks becoming counter-productive. ‘Do you really want me to give you an impact case-study on the back of a day and a half a year?’

But I don’t have easier answers. I’d like to be able to argue that impact is simply one aspect of research, and therefore requires no additional support at all. That would be fairer and more straightforward, but it would also be reckless.

Enhancing impact

The REF doesn’t reward two-star anything. Therefore, as much as some people argue that everyone should be involved in impact, the logic of the system suggests otherwise. If a department doesn’t have a pretty good idea of its 2020ish case-studies now, it’s in a worrying place.

This should clarify matters in a department. Those who are not involved in delivering an impact case-study can – surely should – concentrate singlemindedly on their outputs. Impact, for them, may come into play next time around. For those who are at the impact frontline, meanwhile, attention shifts to enhancement. We have about four years to turn two-star case-studies into four-star case-studies, and there is probably no more important task of REF-preparation than this one.

I’m not sure there are simple answers the question of precisely how we do this. Each case-study is different, and will therefore require different things. The 2020ish winners, I expect, will be the departments that have been flexible and creative in 2015-18, providing bespoke support for developing case-studies.

Documenting impact

How many academics genuinely understand what’s required in an impact case-study? The evidence from REF2014 suggests that many of us don’t. My fear, in this context, is that a lot of activity happening now that may be important in 2019ish is not being adequately documented (and, for that matter, vice versa).

One solution to this problem is fancy new recording mechanisms (which still depend, it has to be said, on academics using them); another is the professional advisors finding work at many places. I wonder whether a further option might be to employ our students to support us. Most departments already spend a lot of money on graduate teaching assistants; why not graduate impact assistants? A smart PGR student, appointed to support a potential case-study, could make a difference: documenting, evaluating, and also enhancing what we’re doing. It would be good work for the students, as well.

———

I don’t have any great knowledge of impact. Indeed, to re-purpose a famous sledge used against the Australian batsman Mark Waugh, I’m not even the best impact-expert in my family. I’m a conventional enough researcher who has rather stumbled into impact work in a couple of projects, and now I find myself paid to worry about it as head of department. I don’t think I’m alone in feeling that we have some work to do.

 

PS. And what is ‘four-star impact’ in the humanities?  See my May 2016 blog-post, based on analysis of departments that achieved 100% four-star gradings for impact in REF 2014.

What’s the difference between ‘game-playing’ and ‘strategizing’? More on the REF.

What’s the difference between ‘game-playing’ and ‘strategizing’? My sense, when it comes to the REF, is that strategizing is what one’s own university does, and game-playing is what other universities do. Especially if those other universities did rather well.

But whatever we call it, success in REF2014 depended to an uncomfortable degree on decisions made by research managers, as opposed to work done by researchers. Here, in my third of too many blog-posts on the REF (others here and here), I want to look at three aspects of gaming. Maybe this is just a way of yoking together a few random things I haven’t said in my previous blogs. Maybe, though, there’s a serious point lurking underneath: that the REF is at its least credible when it looks most like a game.

  1. Double-weighting

Double-weighting is one of the great under-told stories of the REF, having a significant influence in the subjects I know best. This was the first time that universities were responsible for requesting double-weighting, so nobody really knew how high the bar would be set. Some places assumed the standard would be high, and so submitted relatively few requests. Others (including Exeter) took the more logical approach: since there was nothing to lose, why not propose anything that looked substantial (including just about every monograph)?

And what happened? The English panel received 506 requests, and turned down a mighty total of one. Indeed perhaps the great question of REF2014, for me, is: what the hell was that one? I really hope it wasn’t my monograph.

As a result, though, places that were conservative in their requests are now counting the cost. The English panel’s report makes interesting reading:

In a significant number of cases, double-weighting was not requested for outputs where this would clearly have been appropriate. In other cases, the option was used inconsistently, or very sparingly. Some institutions chose not to request double-weighting for any of their outputs. This had an appreciable effect on the outputs sub-profile of a number of institutions.

As I read those words, they’re not happy. The REF, on this matter, became too game-like. Next time, to state the blindingly obvious, the panels surely need to be given more authority to make double-weighting judgements themselves.

  1. The impact case-study

There’s been a lot of analysis of the impact of impact, and there’s bound to be a lot more. One study by Seb Oliver, reported in last week’s THE cover-story, demonstrates the disproportionate effect that impact had on overall results. As the THE summarizes the argument: ‘because the scores for impact (and, indeed, environment) typically show a wider variation than for output, they in effect count for more than their nominal weighting in determining the overall scores’. For the English panel, Oliver estimates, impact thus effectively amounted for 28.6% of the overall result.

But the bigger revelation in the THE piece, for me, was quite how many universities employed consultants to write their case-studies. Now, all the precise prose in the world won’t manufacture impact, but one wonders how many places under-sold themselves on account of trying to muddle through the process without the best advice. How about Newcastle? One of my favourite REF statistics is that Newcastle’s English submission achieved 100% four-star for its impact case-studies, whereas Newcastle’s History submission managed 100% of impact at two-star. Might Newcastle History have been able to sell itself a bit better if, say, it had taken advice from colleagues in English?

The goal is surely that decisions on impact are driven as much as possible by objectively verifiable data rather than professional prose. The reality is perhaps that next time we will all rely on consultants, whether internally employed or externally contracted, thereby driving up the overall administrative cost of the REF and handing more ammunition to its critics.

  1. Environment

And will we also be employing consultants to write our environment statements next time?

One unanswered question from REF2014 is: to what extent were the panels’ assessments of environment dependent on metrics (grant income, PGRs), and to what extent were they dependent upon the (REF 5) templates? I think we were all expecting to be able to trace correlations between the metrics and the overall scores; however, in initial reviews these patterns are far from obvious.

The logical conclusion is that departments, at least to some degree, have been rewarded on account of directors of research presenting information in a convincing manner. Others may have suffered: not necessarily because there were no positive things to be said, but because the authors of the statements didn’t quite get what was important. The effects may have been marginal, but I’m left wondering whether we might reasonably expect something more transparent next time: maybe something with clearer, more explicit directions, making absolutely clear to those preparing the statements the criteria on which they would be judged.

—————————————————–

So what does this all mean? Time after time, looking through the results, we might find ourselves saying: ‘Woah, X screwed up on Y?’ And the person who led the process at X might, of course, be feeling rather queasy. I guess my point is that those responses are not only tough on individuals and their departments – generally good people making honest efforts – but also on the process itself. Ideally, it should look rather less like a game.