Four-star impact

I’ve spent some time recently reading four-star humanities impact case-studies from REF 2014. This is possible because some departments achieved a perfect score for impact, so we know that all their case-studies achieved the highest grade.

As we turn our thoughts to 2021 (or thereabouts), and as impact becomes accepted as part of research assessment beyond the UK, it’s worth pausing to ask what lessons can be drawn from this evidence. My analysis is subjective and impressionistic: really just picking out some patterns that struck me, in the light of discussions I’ve had over the years with colleagues. The departments I’m considering are: English at Bedfordshire, Newcastle, Kingston, Swansea; History at Hertfordshire; Modern Languages at Swansea.

 

Individuals or departments?

One hypothesis with which I began was: if we’re trying to succeed with impact, it makes more sense to think about four-star departments than four-star case-studies. In other words, I suspected that departmental cultures were more important than star individuals.

The evidence is equivocal. At some places, it appears that impact is delivered brilliantly by a minority of staff, albeit with appropriate institutional support. That’s probably the way most of us are working, in fact, not least because of the time that impact-oriented work absorbs. But some of the ‘impact templates’, outlining methodical efforts and commitment across a department, are instructive. For evidence of how to get it right, I’d recommend History at Hertfordshire.

 

Impact and engagement

One of the great academic corridor put-down lines these days is: ‘Oh, that’s not impact; it’s just public engagement’. Seriously, you wonder what planet we’re on at times (and note, by the way, an excellent argument against allowing the assessment cart to be put before the impact horse in the THE). Nonetheless, this line has a point. It doesn’t make sense, from the perspective of time-management, for overworked people to be knocking themselves out on activities that they mistakenly believe will feed into an impact case-study.

So we focus on impact – on having identifiable effects on identifiable groups – and scale back on engagement. Right? Well, the evidence suggests much more porous dividing-lines between impact and engagement than many would like to believe. Take the historian preparing an exhibition on the basis of a chance research discovery. The aim will be to engage as widely as possible, but four-star impact may still be the result.

 

Locality matters

In the early days of impact, one theory was that it would favour London universities. Not so. Some of the most compelling work was achieved by universities a long way from the cultural centre, with powerful regional commitments.

Take English at Newcastle. This department, over many years, identified ways of involving itself in the region. One example of this commitment is its partnership with Seven Stories, the national centre for children’s books: a partnership evident in staffing decisions, grant activity, doctoral studentships, and so forth. Or take English at Swansea, where a longstanding, passionate commitment to Welsh literature and culture shines through in two of their three case-studies.

 

Creativity matters

It’s common sense really: creative people are geared towards engaging with the public. This doesn’t mean that all creative practitioners in universities will be delivering in terms of impact, but it puts them in an excellent position to do so. Take creative writers. Publishing a novel, selling some copies, and doing a few literary festivals may not necessarily amount to much in terms of impact. (Although – and we need to stress this over and over – creative outputs did do rather well in REF 2014.) But if someone writes about a topical or controversial subject – for instance, Jackie Kay (Newcastle) writing about her experiences as a child of Nigerian birth, adopted by white parents in Glasgow – this can lead to important and powerful impact.

 

Questions of longevity

It’s commonly observed that the time-lag between research and impact in the humanities tends to be quicker, on average, than in the sciences. But some of the most impressive case-studies are the product of many years of commitment, and bring benefits that will equally span years. For instance, consider the determination of academics in English at Swansea to bring works of English-language Welsh literature into the public domain, via a series of books that is part-funded by the Welsh government and now sits in every school in Wales. That was an extraordinary achievement.

In this context, of course, it will be fascinating to see how the rules for 2021 deal with the question of ongoing impact from case-studies submitted in 2014. Some of the best impact endures, and hopefully this will be recognized.

 

Questions of cost

Impact costs time and money. In many cases it’s funded by research grants, but across the board it’s apparent that the better departments are underwriting, to a considerable extent, staff time and costs. I’ve commented before about the ‘impact industry’: the advisers, professional case-study writers, and so forth. I think there’s cause to be sceptical about all of that, yet there’s also cause to admire the impressive, moving work documented in these case-studies. The challenge for managers is to find ways of ensuring such work is adequately resourced, at a time when academics and their departments are stretched by competing demands.

 

You can’t take it with you

I’ve noted before the effect upon the impact agenda of the rule, in REF 2014, that if an academic switched universities mid-REF cycle, s/he would take his/her outputs, while the university would retain any impact case-study. That skews the value, to the individual, of impact-related work. Why would a university will spend big money to recruit a major impact-star, as opposed to an output-star, on the eve of  a REF?

One rumour about the next REF is that there may be a perverse solution: perhaps outputs, as well as impact, may be retained for submission by previous employers. Well, let’s wait and see. Such a move might well help keep a lid on wage inflation at the top end; although the implications for early-career appointments would need consideration.

Advertisements

REF-cycling

One of the characteristics of the REF/RAE is that we always spend two or three years playing a game with no goalposts. It is now over two years since the deadline for REF 2014 submissions. It will be close to three years by the time we know what form REF 2020-something will take.

This hiatus poses a question: to what extent should we plan when we don’t know the rules? Responses might be positioned on a spectrum. I’ve heard of departments where academics are told to keep calm and carry on doing good research. There’s a simple decency to that. And there are others where plans, drafts and publications are being reviewed, monitored and graded, more assiduously than ever.

My university is somewhere towards the latter end of the spectrum. That’s just the way we are. And I feel myself, at the level of a department, that it would be irresponsible not to ensure that colleagues are carefully assessing the possible consequences of decisions about research plans. Research time is precious – and REF contributions, let’s face it, have a huge bearing on career progression in the UK – so these decisions will matter.

That’s why we’ve been spending some time recently, in my department, listening to REF 2014 panellists and thinking about our planning. I’ll outline here – in a post that is admittedly humanities-heavy and UK-centric – some of what’s been on our minds.

 

Originality, significance, rigour

The REF criteria are simple enough, attuned with what most of us value about academic work. (For some helpful expansion, see the Panel D criteria, p. 88.) Originality, significance, originality: and ‘significance’, it’s worth noting, should not be confused with ‘influential’, since a publication may be significant, in the eyes of a REF panel, even in a small area and with a small readership. (That’s one way metrics could change things.)

In my department, though, one of the lessons we’ve been learning is that we’ve relied too much in the past on proxies of quality. Most of all, we’ve been overly swayed by length. In our REF 2014 planning, we made easy assumptions that monographs would be graded four-star. Now we’re accepting that there were plenty of three-star monographs submitted to the English panel in 2014, and even some two-star ones. Meanwhile, there were plenty of articles graded at four-star: something we were equally reluctant to predict. (Equally, plenty of editions, collections of essays, creative works: there are some myths about what ‘matters’ in the REF that are well worth exploding.)

What lessons might we draw from this? Firstly, it seems to me there are risks of over-reacting, or of seeing the positive message (‘my articles could be four-star’) and missing the negatives (‘my monographs might not be’). Secondly, there are some really interesting mentoring challenges, because if I have colleagues devoting a lot of time to writing monographs, I want to do all I can to help them ensure those monographs are as good as they can be. The alternative might be deciding, in a future mock-REF process, that some monographs don’t cut it. And that would be a tough, potentially divisive message.

Thirdly, it perhaps changes the mind-set, quite common in the humanities, that goes along the lines: ‘focus on the monograph and sort out some other items along the way’. If an exceptional article may rate better than an unexceptional monograph, we need to be thinking very carefully about all our items.

Fourthly – and this isn’t strictly relevant, but I like to say it whenever I can – we need to think about what not to do. There’s no point showing up for a REF with a pile of two-star publications and a case of work-related stress.

 

Game-playing or dodgy rules?

There was a lot of talk about ‘game-playing’ after REF 2014, much of which was sour-grapes silliness. But it’s becoming pretty clear that some of the rules created the effect of game-playing, regardless of the players’ intentions.

Exhibit A is ‘double-weighting’. Asking units of assessment to decide whether or not to request double-weighting for an item distorted the playing field, hurting those who interpreted the requirements more conservatively than others. The solution next time must be to give panels more authority to determine what is in the best interests of the submission.

Then there’s ‘repetition’: the submission of work that had previously been published in another form. We were appalled to be criticized for this, since we had been very careful; however, it now seems rather a lot of English departments received the same feedback. The application of the rules, let’s say, had a punctilious vigour. I’d hope we can do better next time – but in the meantime, it pays to know.

 

The impact industry

Allow me to quote again my favourite sentence from the Green Paper: ‘We must also address the “industries” that some institutions create around the REF and the people who promote and encourage these behaviours.’ And hence to impact.

The impact industry is building. There are impact gurus who will help us maximize our impact, case-study writers to support us as we prepare for a REF, and training programmes to help us understand it all. My impression is that the level of expertise around impact in the humanities is rather less reliable than it is for the sciences and social sciences. Although I’ve made this point before, it therefore remains crucial to nurture and respect experience – knowledge of what works, and what was recognized in 2014 – within our disciplinary communities.

Thinking ethically, one wonders whether all the money devoted to the impact industry might be better spent. But thinking rationally, this was surely bound to happen. I mean, what did they expect? There was indeed a touch of utopianism to that Green Paper sentence.

 

Might the rules all change? I wouldn’t bet on it. I’m predicting a REF 2021 that will look rather like REF 2014: with maybe a few of the creases ironed out, and probably a few more ironed in. But I might be wrong.

You can’t take it with you: impact 2020

Here’s a pre-REF 2020ish scenario. Professor X and Professor Y get job offers elsewhere. Professor X has two four-star publications, Professor Y has prepared two four-star impact case-studies. The university wants to retain them both, but funds are tight and priorities need to be clear. The impact case-studies are worth eight times as much as the publications, but publications travel with their authors whereas impact case-studies (under 2014 rules, at least) are owned by the university. So Professor Y goes; in fact, s/he probably wouldn’t have had a job offer at that time in the first place.

Hence the response of one of my colleagues, reviewing an application I’d prepared for the AHRC follow-on (i.e. impact-related) scheme. ‘This is all lovely,’ he said, ‘but what’s in it for you?’ He’s not an impact Luddite, mind you; his response was that of someone who has been ploughing this furrow longer than I have, and occasionally wondering why.

I don’t want to argue for more money for the impact-active. The real problem, it seems to me, is the way outputs distort the job market, not the way impact fails to do so. But I suspect that a lot of us who are in the business of managing departments are still grappling with the question of how to plan for 2020ish. How do we best recognize quality impact? How do we support and enhance potential case-studies? And how do we document them?

Recognizing the impact stars

To return to the scenario of my opening paragraph: in actual fact now, not 2019, is the time to buy impact. In the humanities, we learned in REF2014, lead times between research and impact tend to be shorter than for the sciences, meaning that it’s perfectly plausible to employ someone in September 2015 and expect a four-star impact case-study from them four years later.

Are any universities doing this? Not many, after all the pre-REF spending, will have sufficient space in their business plans. There have been a couple of ads for chairs in digital humanities – which is interesting in terms of the growth of a field, and perhaps also suggests a perception that this is an impact-rich field – but I haven’t seen any bold moves more specifically in my own discipline. Some of the more interesting decisions may fall at the lower end of the scale. When looking at applicants for lectureships, how important might it be that a candidate has the capacity to add value to a planned case-study? Might this prompt us to think, in the humanities, a little more than we have traditionally done in terms of research teams?

And how do we recognize those already delivering impact? One question of principle, which has a bearing on workloads and recognition, is whether impact is simply one aspect of research, or something else again. Hence, when allocating workloads it’s not clear whether we expect someone to deliver impact as part of their allotted research time, or whether we need to create more space for the impact-active. We’re still fudging this in my department. For this year I was granted, to support my impact work, ten hours in our 1650-hour annual workload model. That’s a figure so ‘neither here nor there’ that it risks becoming counter-productive. ‘Do you really want me to give you an impact case-study on the back of a day and a half a year?’

But I don’t have easier answers. I’d like to be able to argue that impact is simply one aspect of research, and therefore requires no additional support at all. That would be fairer and more straightforward, but it would also be reckless.

Enhancing impact

The REF doesn’t reward two-star anything. Therefore, as much as some people argue that everyone should be involved in impact, the logic of the system suggests otherwise. If a department doesn’t have a pretty good idea of its 2020ish case-studies now, it’s in a worrying place.

This should clarify matters in a department. Those who are not involved in delivering an impact case-study can – surely should – concentrate singlemindedly on their outputs. Impact, for them, may come into play next time around. For those who are at the impact frontline, meanwhile, attention shifts to enhancement. We have about four years to turn two-star case-studies into four-star case-studies, and there is probably no more important task of REF-preparation than this one.

I’m not sure there are simple answers the question of precisely how we do this. Each case-study is different, and will therefore require different things. The 2020ish winners, I expect, will be the departments that have been flexible and creative in 2015-18, providing bespoke support for developing case-studies.

Documenting impact

How many academics genuinely understand what’s required in an impact case-study? The evidence from REF2014 suggests that many of us don’t. My fear, in this context, is that a lot of activity happening now that may be important in 2019ish is not being adequately documented (and, for that matter, vice versa).

One solution to this problem is fancy new recording mechanisms (which still depend, it has to be said, on academics using them); another is the professional advisors finding work at many places. I wonder whether a further option might be to employ our students to support us. Most departments already spend a lot of money on graduate teaching assistants; why not graduate impact assistants? A smart PGR student, appointed to support a potential case-study, could make a difference: documenting, evaluating, and also enhancing what we’re doing. It would be good work for the students, as well.

———

I don’t have any great knowledge of impact. Indeed, to re-purpose a famous sledge used against the Australian batsman Mark Waugh, I’m not even the best impact-expert in my family. I’m a conventional enough researcher who has rather stumbled into impact work in a couple of projects, and now I find myself paid to worry about it as head of department. I don’t think I’m alone in feeling that we have some work to do.

 

PS. And what is ‘four-star impact’ in the humanities?  See my May 2016 blog-post, based on analysis of departments that achieved 100% four-star gradings for impact in REF 2014.

What’s the difference between ‘game-playing’ and ‘strategizing’? More on the REF.

What’s the difference between ‘game-playing’ and ‘strategizing’? My sense, when it comes to the REF, is that strategizing is what one’s own university does, and game-playing is what other universities do. Especially if those other universities did rather well.

But whatever we call it, success in REF2014 depended to an uncomfortable degree on decisions made by research managers, as opposed to work done by researchers. Here, in my third of too many blog-posts on the REF (others here and here), I want to look at three aspects of gaming. Maybe this is just a way of yoking together a few random things I haven’t said in my previous blogs. Maybe, though, there’s a serious point lurking underneath: that the REF is at its least credible when it looks most like a game.

  1. Double-weighting

Double-weighting is one of the great under-told stories of the REF, having a significant influence in the subjects I know best. This was the first time that universities were responsible for requesting double-weighting, so nobody really knew how high the bar would be set. Some places assumed the standard would be high, and so submitted relatively few requests. Others (including Exeter) took the more logical approach: since there was nothing to lose, why not propose anything that looked substantial (including just about every monograph)?

And what happened? The English panel received 506 requests, and turned down a mighty total of one. Indeed perhaps the great question of REF2014, for me, is: what the hell was that one? I really hope it wasn’t my monograph.

As a result, though, places that were conservative in their requests are now counting the cost. The English panel’s report makes interesting reading:

In a significant number of cases, double-weighting was not requested for outputs where this would clearly have been appropriate. In other cases, the option was used inconsistently, or very sparingly. Some institutions chose not to request double-weighting for any of their outputs. This had an appreciable effect on the outputs sub-profile of a number of institutions.

As I read those words, they’re not happy. The REF, on this matter, became too game-like. Next time, to state the blindingly obvious, the panels surely need to be given more authority to make double-weighting judgements themselves.

  1. The impact case-study

There’s been a lot of analysis of the impact of impact, and there’s bound to be a lot more. One study by Seb Oliver, reported in last week’s THE cover-story, demonstrates the disproportionate effect that impact had on overall results. As the THE summarizes the argument: ‘because the scores for impact (and, indeed, environment) typically show a wider variation than for output, they in effect count for more than their nominal weighting in determining the overall scores’. For the English panel, Oliver estimates, impact thus effectively amounted for 28.6% of the overall result.

But the bigger revelation in the THE piece, for me, was quite how many universities employed consultants to write their case-studies. Now, all the precise prose in the world won’t manufacture impact, but one wonders how many places under-sold themselves on account of trying to muddle through the process without the best advice. How about Newcastle? One of my favourite REF statistics is that Newcastle’s English submission achieved 100% four-star for its impact case-studies, whereas Newcastle’s History submission managed 100% of impact at two-star. Might Newcastle History have been able to sell itself a bit better if, say, it had taken advice from colleagues in English?

The goal is surely that decisions on impact are driven as much as possible by objectively verifiable data rather than professional prose. The reality is perhaps that next time we will all rely on consultants, whether internally employed or externally contracted, thereby driving up the overall administrative cost of the REF and handing more ammunition to its critics.

  1. Environment

And will we also be employing consultants to write our environment statements next time?

One unanswered question from REF2014 is: to what extent were the panels’ assessments of environment dependent on metrics (grant income, PGRs), and to what extent were they dependent upon the (REF 5) templates? I think we were all expecting to be able to trace correlations between the metrics and the overall scores; however, in initial reviews these patterns are far from obvious.

The logical conclusion is that departments, at least to some degree, have been rewarded on account of directors of research presenting information in a convincing manner. Others may have suffered: not necessarily because there were no positive things to be said, but because the authors of the statements didn’t quite get what was important. The effects may have been marginal, but I’m left wondering whether we might reasonably expect something more transparent next time: maybe something with clearer, more explicit directions, making absolutely clear to those preparing the statements the criteria on which they would be judged.

—————————————————–

So what does this all mean? Time after time, looking through the results, we might find ourselves saying: ‘Woah, X screwed up on Y?’ And the person who led the process at X might, of course, be feeling rather queasy. I guess my point is that those responses are not only tough on individuals and their departments – generally good people making honest efforts – but also on the process itself. Ideally, it should look rather less like a game.

How’s My Impact?

A lot of my time over the past months, when I haven’t been involved in head of department stuff, has been occupied by the question of impact. This is an interesting, and not entirely expected, by-product of my role as Principal Investigator on two AHRC-funded projects. The question one now faces, along with more predictable concerns about academic outputs, is: will this project produce an impact case-study?

Consequently, I’ve been prompted to think about reseach in new ways, and catapulted into meetings with all sorts of people I never expected to meet, speaking all sorts of alien, non-academic languages. There’s been lots of talking and a fair number of dead-ends; inevitably, also, there will be more grant applications to complete. In all, it’s a little bit like middle-management, only more fun.

This is new territory for many of us, since the era of supporting impact-development is really very young. Given that impact generally needs to be tracked back over a period of years in order to count, REF2014 may look, in retrospect, anomalous. It was the REF for which we had to make sense, as well as we could, of things we had been doing without realizing they actually mattered. (For 2014 definitions, by the way, see s.D3 at: http://www.ref.ac.uk/media/ref/content/pub/panelcriteriaandworkingmethods/01_12_2D.pdf.)

But future REFs will be different. And this situation prompts a number of questions: from which, here, I will choose four.

 

  1. Do we all need to be impactful?

We’re likely to need more impact case-studies in 2020. In 2014 we needed one for every six researchers whose work was submitted. I’d bet we won’t be looking at 1:3 next time, as some have predicted, but 1:5 is likely and 1:4 possible.

That means many, or even most, of us will be involved in impact. Many case-studies, after all, are configured as the product of research groups or joint-run projects, rather than the work of lone-impacters. But we also need to be sensible. This suits some people more than others, and we need to recognize the life-cycles of researchers and their projects. Asking an early-career researcher to be impactful before s/he has finished a monograph may be counter-productive for everyone. And we need to remember that impact activities without the underpinning publications may well be ‘public engagement’, but may not actually be ‘impact’. So carts and horses need to be properly aligned.

 

  1. How can we accelerate our impact?

A number of research councils, including the ESRC, are awarding big ‘impact-accelerator’ grants to selected universities. This money is then broken down internally, to support the consolidation – acceleration, even – of impact in particular areas. That will provide a competitive advantage to the universities involved: another instance, if you like, of research councils furthering the process of research concentration.

The AHRC has no comparable plans. It stands resolutely against the principle of research concentration, even as the figures suggest that it’s happening all the same. But its follow-on scheme, open to all existing grant-holders, provides tens of thousands of pounds for impact-focused work, and the success rates for this scheme are notably high (around 50%). Obviously, also, any project or fellowship application will now have an element of (fundable) impact activity. Hence the question I’ve been asking myself: any grant, and any grant application, provides an opportunity to produce an impact case-study somewhere down the line. It’s not always going to work, but we’d be irresponsible not to consider the possibilities.

And what might be done internally? ‘Research leave’ in our college of humanities has been relabelled ‘research and knowledge-exchange leave’. Well, I can’t see that catching on among the troops (‘I’m really enjoying my researc and knowledge-exchange leave’), but the principle is sound enough. If impact matters for the REF, it’s worth supporting in all the ways in which we support other research activity.

 

  1. How do we record our impact?

One of my questions to my director of research centres on Professor X, who is generating activity that could produce a wonderful case-study in 2020, but who is pissed off with his employer and rather expecting to be elsewhere five years from now. Given that impact, in HEFCE’s definition, doesn’t travel with the individual (and given also that Professor X may decide in due course that the grass is actually no greener elsewhere), we need to be keeping track of what s/he is up to. How do we do that?

I’m not sure we have a good answer to that question. I’m not Professor X (well, maybe a little bit – but that’s another blog), but I have no system whatsoever for documenting what’s happening on the wonderfully impactful ‘Poly-Olbion Project’ and its HLF-funded sister-project ‘The Children’s Poly-Olbion. In fact, some of the best activity happens almost without me noticing – as, perhaps, is the nature of the most effective impact-generating projects.

So what do we do? Pay student interns, perhaps, to document what we’re doing? I still like that idea, although I tried it last year and it’s not as straightforward as it sounds. Give Professor X some impact-hours in the workload model? We’re trying this now and – well – it’s a nice gesture but won’t necessarily produce the desired behavioural change in Professor X. An online record-keeping system? Will Professor X choose to spend his evening hours with that?

 

  1. How should we reward impact?

If impact can’t be packed in the luggage when a researcher moves, what incentive does an employer have to reward impact-related work? In 2019 we’ll doubtless find Professor Y, with a REF-portable four-star monograph, getting the job offers and pay-rises, as has always been the case. But Professor X?

This seems a key systemic issue, and I’ll be interested to see whether HEFCE tackles it when it rethinks the position of the 2020 goalposts. But it’s also an internal one: a question for managers and promotion committees, in particular, and one still to be answered in a convincing way.

But things are changing, just as our ideas of research are changing.