Not much love for the TEF*

How do we like the Teaching Excellence Framework so far? According to a survey of participating higher education institutions conducted earlier in 2017 – after documents were submitted but before results were announced – the answer looks like a resounding: ‘not much at all, thanks’. This survey, part of a rapid-response Universities UK review of the TEF, poses some important questions, centring particularly on the relation between future costs and benefits.

The survey results are heavy with the collective shrugging of PVC shoulders. Asked whether ‘The TEF will accurately assess teaching and learning excellence’, 73% of respondents actively disagreed while just 2% agreed. Given that only 83 universities responded, that means the managers of only two universities in the entire country are prepared to argue that the TEF does its job. Or, to be more accurate, that was the case before the feel of a gold medal possibly changed some minds.

But there is also evidence of a curiously widespread acceptance of TEF. Asked whether it will make a ‘positive contribution to student decision-making’, only 18% agreed. Asked whether it will ‘enhance teaching and learning practice’, views were roughly split (25% agree; 29% disagree). But asked whether it will ‘enhance the profile of learning and teaching’, 73% agreed. It may be unreliable, even misleading – seriously, overall these are awful results, far below my expectations – but at least it’s putting education on the agenda.

It’s perhaps also driving spending decisions within universities. Asked about investment subsequent to the announcement of TEF in 2015, the general attitude seems to be: ‘well, you see, we were going to do that anyway’. Indeed 53% said TEF had no influence or impact on such decisions. But the raw facts of investment in learning and teaching in this period are nonetheless impressive: ‘additional investment in academic support’ (over 70%); ‘additional investment in learning and teaching’ (over 80%); ‘new progression routes for teaching’ (over 20%).

There are plenty of critics of UK higher education who are asserting, on the back of frankly bugger all evidence, that universities are not investing along these lines at all. We’re spending it all on vice-chancellors’ salaries, their houses and, like, loads of other fat-cat stuff. Maybe these critics would say: well, the universities would say they’re doing all that worthy stuff, wouldn’t they? But it’s actually dead easy to document this investment, if anyone cares to look; in fact those provider submissions will help. Our critics are fixated on the lack of competition on price, but anyone working in higher education knows that there is intense competition at the level of quality.

Those, including me, who would argue from this point that the TEF is too important to be allowed to fail might also derive comfort from the estimated cost. Taking account of estimates of staff time devoted to the 2017 TEF by universities, the cost came in at about £4.1 million. Now I know what you’re thinking: we could employ at least nine extra fat-cats for that money. We could let them loose to roam wastefully around the country. But it’s worth remembering that the REF comes in at £212 million. If the TEF helps universities, in the face of ill-informed criticism, to demonstrate commitment to learning and teaching, maybe it’s a steal at £4 million.

But I sense a dilemma. The devastating lack of confidence in the process must be addressed, since indifference could so easily lead to cynicism, and from there to notoriety. Hence the attention that will be devoted to responses to the questions concerned with ‘priorities for future development’. There’s not exactly unbridled enthusiasm here, but let’s work with what we’ve got. Among other things, we have some interest in measures of ‘learning gain’, a fairly solid commitment to the value of teaching qualifications, and a vague appeal to the idea of ‘new metrics’ and ‘more sophistica[tion]’.

Sophistication and new metrics, however, sound kind of expensive. Measuring learning gain is a bottomless money-pit. And subject-level TEF, to which the government remains committed in the face of an overwhelming lack of interest within the universities, is predicted to cost up to £13.7.

So what’s the value to the sector of a TEF that will command a greater degree of confidence? One of the intriguing facts offered in this report is that there was a high correlation between Guardian league table results – which cost the sector nothing – and TEF results. I think I might have predicted that would happen; although if I did, I’ll concede that it was always likely to be the case. Consequently, it seems to me self-evident that if TEF in future is to be demonstrably better than existing league tables, and certainly if it is to be used as the basis for differentials in fees, it will inevitably become significantly more expensive. At what point will its value justify its cost?

* Published under a different title by wonkhe.com

Advertisements

‘We need to talk about your gross teaching quotient’*

There’s a new frontier in the quest to demonstrate teaching excellence, and it’s called the Gross Teaching Quotient. By mid-winter senior managers will be descending upon departments across the country bearing spreadsheets and bad news: ‘We need to talk about your Gross Teaching Quotient’.

It’s gold, gold, gold for the English Department

The Gross Teaching Quotient – and let’s call it GTQ, because that’s what it will become – is the most eye-catching aspect of the recent proposals outlining the subject-level Teaching Excellence Framework. It’s just a pilot version at this stage, but it’s coming at us fast, scheduled to be in place for real in 2019-20.

I’m broadly supportive of institution-level TEF. Some aspects of it, such as the olympic-style medals, are manifestly barking, but there’s no avoiding the push to demonstrate teaching excellence. The TEF uses reasonable proxies for quality – and I haven’t heard any better suggestions, despite all the noise – while the written submission encourages innovative actions and reflective thinking.

But I’ve had my head firmly in the sand on subject-level TEF. On practical grounds the concept has seemed too cumbersome, while I’ve also wondered about the relation between cost and value. I mean, if institutional-level TEF is already focusing managerial minds and driving reform – as I think it is, though nobody has bothered to wait long enough to assess this – where is the added value to subject-level TEF?

Jo Johnson’s answer would be that it’s all about the consumer. Potential students generally choose courses first and universities second, so they will want evidence of quality at that more granular level. In my experience such people are in fact overwhelmed with evidence – from league tables, unistats and so forth – but perhaps that might equally support Johnson’s argument. Some people, it seems, just want to see a gold medal.

The pilot will include only a handful of universities. Subject-level TEF will use the same metrics as institution-level TEF, with a few notable additions. There will be a written submission, stripped back to five pages. And there will be 35 subjects or subject-groupings, to avoid the metrical muddle that can be caused by small disciplines.

There are two pilot models. Model A (‘by exception’) will simply give subjects the same rating as their institutions unless the metrics indicate a need for closer investigation. Model B (‘bottom-up’) will assess each subject fully, and build towards an institution-level award from this basis. I propose to label these, respectively: ‘the sane model’ and ‘brace yourself, it’s coming’.

Then it starts to get interesting. All the well-meaning complaints from across the sector about the TEF merely measuring proxies have got the TEF-team thinking. They’re not for turning; they’re marching ever onward to the holy grail of quantifiable teaching excellence. And this brings them – as, of course, it would – to the GTQ.

The GTQ is a measure of ‘teaching intensity’. And teaching intensity is not all about contact hours; honest, the document says this maybe fifteen times, so I swear they must mean it. Instead it’s an idea, kicked about in last year’s Success in a Knowledge Economy White Paper, about the relation between the quantity and quality of teaching. And that was all derived from Graham Gibbs’s 2010 report Dimensions of Quality.

Teaching intensity will be measured in part by a student-survey. Think about that for a minute: students will be asked questions about whether they’re getting enough teaching. And then there will be the calculations, weighting ‘the number of hours taught by the staff-student ratio of each taught hour’. Got it? The GTQ is then ‘calculated by multiplying the taught hours by the appropriate weighting and summing the total across all groups, followed by multiplying by 10 to arrive at an easily interpretable number’. And then it’s divided by the square-root of staff days lost due to stress. Or something like that.

It’s worth noting just what a narrow reading of Gibbs this actually is. In the desperation to create a new metric, lots of valuable Gibbsian ‘dimensions’ have been set aside: from the critical questions of who does the teaching and how well they have been trained (puzzlingly ignored by TEF so far), through assessment and feedback, and beyond. I guess we get to brush up on this stuff when we’re preparing our written submissions, but there’s a curious narrowing of vision for all the rhetoric to the contrary.

My greatest concern about all this is that GTQs will evidently be produced as comparative measures, driven by an underlying assumption that more intensity is always going to be better. Practice in my department may be perfectly sound from all sorts of perspectives; however, as I understand the proposals, if our GTQ is weaker than a competitor’s we may be heading for silver. Admittedly  it’s only one metric, but from a management perspective it will attract attention as the newest and perhaps the easiest to manipulate. Subject-level TEF will understandably instil anxiety in all sorts of people in management positions; not all of them can be relied upon to respond reasonably.

This is a pilot, and much may change between now and full implementation. Crucially, the review of the first round of institution-level TEF will affect anything that happens thereafter at subject level. But subject-level TEF may affect departments pretty much immediately. Much of that change will be for the good, some will be more questionable, and an awful lot could increase workloads and stress-levels, for students and  staff alike.

* This piece was first published in Times Higher Education, 10/8/17

But only one tantrum: getting it right and wrong on TEF results day*

The TEF results have come and gone, and the press has predictably declared some of the nation’s best universities to be ‘second-rate’. One lesson to be drawn from the past few months is that there are plenty of people determined to kick the British university system for no better reason than that it remains world-class despite an overriding spirit of national decay. The TEF has fed such commentators an easy line.

But setting aside those frustrations, what might we learn from initial responses across the sector? What might they tell us about the faultlines in debates over TEF as we move forward?

 

Only the one tantrum? Really?

Just a hunch, this, but I reckon that when Christopher Snowden, VC of Southampton, cut loose within hours of his university’s bronze award being announced, he rather expected to be leading a chorus. A concerted vice-chancellorial spitefest might have battered a hole in the TEF.

Tantrums there must surely have been, but Snowden is to date the only one (as far as I know) to have taken his into the public domain. I suspect he’ll regret it. Meanwhile there are some outstanding examples of how to present mediocre news. Bristol, for instance, takes the opportunity to boast about other league table successes, and lists some ‘ongoing improvements’. UCL’s Michael Arthur boldly concedes that there remains another level beyond silver: ‘UCL puts education firmly on a par with research and we will not be satisfied until we have achieved a gold standard’.

Maybe those statements are easier to articulate when one’s institution has outperformed expectations, but their focus on the future is smart. In my mind that’s also preferable to the vainglorious puff produced by some of the golden ones. But perhaps the dominant impression, certainly from non-gold Russell Group universities, is a desire that this whole bloody thing will just go away. Some don’t even mention the TEF on their websites. Others waited a day or so to do so, when they had had the information two days before its release. Listen carefully: that’s the sound of institutional passive aggression.

All of which suggests that the TEF has roughly the same level of security as Brexit. It will probably survive, but plenty of people will try to soften it or blow it out of the water in the months and years to come.

 

But when did words ever matter?

What, then, do responses indicate about the possible shapes TEF might assume? For me, one surprising aspect of the post-results commentary has been criticism  that provider submissions have influenced the panel’s judgement. In other words, a set of metrics that indicated a lower grade has in some instances been trumped by sixteen pages of prose – or, in a couple of PVC-career-wrecking cases, vice versa.

Goddamn, the university system in this country is addicted to metrics. We’ll bicker forever and a day about what those metrics should be, then we’ll cry foul when they aren’t followed slavishly by the experts we’ve put in place to exercise some discretion. Why not just use a computer next time?

I’d argue, on the contrary, that those submissions are a critical part of the process. When I was involved in producing my university’s statement, I was vaguely aware that it probably wouldn’t much matter since our metrics were already good. Yet, as I’ve written before, the process of that document’s composition, which involved senior figures across the university, was part of the TEF’s point. It focused minds on what we were doing well, and what we could be doing better. Now we’ll work through the submissions of other successful universities, looking for further ways to improve.

Anyone who argues that these documents should be disregarded doesn’t understand the relation between self-reflection and transformation. But it seems that this point might need to be made to metrics-heads in reviews to come. Moreover, those arguing that a gold rating will lead to complacency similarly fail to understand how such levels of performance have been achieved. My university’s gold award is the result of a decade or more of relentless attention to the student experience. Nobody can expect to slack off for a couple of years and revive things in time for the next assessment.

 

Future medalling

I think I’ve made my disdain for the olympic-style medals clear enough already. But I appreciate the dilemma. The architects of TEF wanted something that couldn’t easily be brushed aside. They also wanted to avoid the silliness of meaninglessly fine league-table gradations, of the kind produced by the National Student Survey. Chris Husbands, the TEF Chair, has been quick to rubbish league tables produced on the basis of the TEF results.

It seems to me that some of this year’s silver medallists might have pointed the way to the future in their press releases. Warwick’s headline, for example, states simply: ‘Government declares Warwick teaching “excellent”’. And so it did. By implication it declared certain other universities to be ‘outstanding’. Perhaps those bronze universities, meanwhile, were ‘good’. Maybe that sounds a bit OFSTED-ish, but there’s a value in that kind of recognition.

One interesting question, though, is grade-inflation. A system of medals suggests there will always be a decent proportion of bronzes. But what if TEF does its job and raises levels of performance? Will panels of the future reward those improvements, or systematically raise the bar?

 

I’m betting the TEF will continue for a while, though it’s bound to change. Focusing on proxies of teaching quality, though unpopular with some, remains the only feasible method. Medals remain irredeemably mad. Subject-level assessment looks as cumbersome and impractical as ever, and might quietly be shelved. Linking the right to raise fees to a silver rating or gold rating, and not messing around with gradations between them, looks like a sensible modification of the crazily finely-grained model originally proposed. And honestly, even the LSE might be able to get a silver if they set their minds to it.

* Published under a different title by wonkhe.com

What next for the TEF?*

The fact that the first Teaching Excellence Framework results will be published the week after the election wasn’t planned that way, but feels appropriate. Somehow it seems as though that the entire process of TEF-construction has been conducted in the shadow of a dysfunctional political system with other things on its mind.

Indeed, looking back on the hurried, slightly chaotic passing of Higher Education and Research Bill in the days after the election was announced, I wonder whether we will this as the moment when the TEF started to assume rather more docile a shape than we had been led to expect. The review of the TEF, scheduled to happen ‘by the end of 2019’ – in other words, about as soon as possible – looks ominous.

So here, having followed the TEF through its development (like, here and here, and a bit more here), and having got a couple of predictions right already, I’m going to make a few more. I have no evidence for my claims; I just reckon I might be right.

 

Gold, silver, bronze: bin

This is the easy bit. The ‘gold’, ‘silver’ and ‘bronze’ rankings, announced in the euphoric wake of Australia’s (or maybe that was Great Britain’s – but surely not) medal haul in 2016, is universally accepted as bonkers. It belittles the whole process of assessing teaching excellence, and it unreasonably stigmatizes universities that are in actual fact doing a perfectly good job.

So this will go, as soon as it can be shuffled off stage in a seemly manner. I’d even bet that people in the Department of Education – maybe even the halfwits who dreamt it up in the first place – have discussed whether it might be ditched for this round, given the way that it undermines the image of the entire TEF enterprise. But that possibility looks just a little implausible, even in these turbulent times.

That doesn’t necessarily mean, however, that it would revert to becoming a pass/fail exercise, like a parallel version of a QAA review. I expect there will be gradings of some kind, and that these will be fed into existing league tables; however, they will be more nuanced – and, frankly, also more useful – rather on the model of REF results. In this way assessments will provide objective and constructive information, both for universities and applicants, without being unnecessarily damaging.

 

TEF results linked to ‘fee-rises’: believe it when you see it

So TEF results were all set to be linked to ‘fee rises’ (albeit that these were never rises at all, merely – at best – increases in line with inflation). But here’s me back in November last year:

One question I’ve been asking since the TEF was first proposed goes roughly along the lines: what happens when people at the highest levels of government realize that some of the universities most likely to lose from the exercise are some of the country’s biggest higher-education brands?

I must admit that I never equated the House of Lords with ‘the highest levels of government’. I rather overestimated the capacity for people elsewhere to behave sensibly, and underestimated the influence of this curious remnant of British elitism.

But it happened. In my imagination it was just a matter of a couple of VCs of high-profile universities that were looking down the barrel of a ‘bronze’ sitting down to dinner with a few inluential alumni in the House of Lords and explaining the outrage about to unfold. Their grand alma mater will be forced to charge lower fees than the scruffy post-92 down the road. Really. Perish the thought.

And so, one way or another, I’d bet this will happen again. The sensible compromise option – submit yourself to TEF, get a satisfactory rating (i.e. anything from bronze upward, in this round’s terms), and you can raise your fees in line with inflation – seems to me like a long-term solution. Such an outcome would also help to revive the battered reputation of the National Student Survey, on the back of the National Union of Students’ boycott. (That worked a treat, didn’t it guys?) Maybe those running the TEF might even see their way to promoting the NSS, rather than questioning its value.

 

Subject-level TEF: yeh, right

Sorry, but I just see the prospect of subject-level TEF as madness. I was involved in preparing Exeter’s institutional response, which was hard work but served a purpose, usefully prompting us to reflect on what we’re doing. Moreover, when the institutional statements are all published, universities across the land will have at their fingertips a huge bank of ideas for educational innovation. That’s great for everyone, most of all the students.

But what’s to be gained from a subject-level exercise? In sheer bureaucratic terms, I can’t see how the labour – on the part of the academic departments or the reviewers – will justify the outcomes. In financial terms, I can’t realistically see how results could be factored into course fees, creating a weird patchwork of slightly different fees across any university. And – most importantly – as an exercise in driving forward innovation I’m not sure this will add in any credible way to the achievement of the institutional exercise.

 

Get a future; get a logo

Finally, the TEF is suffering a serious branding deficit. It needs a logo. It needs some pictures on its website, for God’s sake. Everything about it screams: ‘bleak and bureaucratic’. It sits up and says, ‘Hate me’. I’m predicting this might change; although, to be honest, I’m less confident about that than any of the above. In the meantime, for the sake of an image for this blog-post, I offer you a kitten (nicked shamelessly from the web).

 

So that’s all settled, then. What could possibly go wrong from here?

* This piece has been republished by wonkhe.com. But they didn’t publish the kitten. I mean, seriously, it’s only me giving you the cuteness factor.

Time for a student-support arms race: entering the era of the Teaching Excellence Framework

Here’s a coincidence. Over the past couple of weeks, senior academics and managers across the UK have been polishing their ‘provider submission’ documents for the first cycle of the Teaching Excellence Framework. If you haven’t felt the terror of thistelegraph-final process, you just haven’t been in the right meetings.

Meanwhile up north, the University of Manchester has become the latest to advertise new paid posts for ‘working-class officers’. This kind of appointment, through the students’ union, won’t cost the university very much money but is an important statement of a perceived need.

I think these two things are – indirectly, at least – related. Staring into the abyss of a possible ‘silver’ – or, God forbid, ‘bronze’ – rating rather concentrates the mind. It might quite logically lead to new initiatives and commitments. Indeed I’m prepared to ask here whether – contrary to the assertions of those inflation-deniers at the National Union of Students – the TEF might turn out to be rather a good thing for students after all.

 

Cultural diversity and all that

The Manchester initiative follows similar appointments at SOAS and St Hilda’s, Oxford. While some will doubtless mock the idea that universities need these posts, they seem to me realistic and responsible responses to emergent needs.

There is ample evidence that students are arriving at university with more complex and urgent requirements for support than ever before. Wellbeing and disability services are oversubscribed across the country, while freshers arriving with weak qualifications often need intensive study-skills support if they are to adjust successfully to university-level study.

Cultural diversity presents further challenges. As I’ve written before, it can be all too easy for universities to assume that they’ve done their bit for widening participation simply by getting applicants from marginal and disadvantaged groups through the doors. But at that moment the challenges for those students are only just beginning, especially if they’re arriving at a university where the vast majority of students are middle class and white.

The TEF, by the way, recongnizes this issue. The bundle of data handed to universities includes figures on retention, broken down by categories of students. If a university is failing to support its non-standard students, it will be exposed.

 

The TEF and the art of bullshit-detection

The institutional TEF submission this time around consists of that institutional data-set and a supporting document. The data will give each university a pretty good indication of how they will be graded, but it seems that the written submission may make a moderate-to-significant difference, especially in borderline cases.

Universities have not been given much guidance about how to approach the provider submission. But one of my conclusions, based on the experience of a long day editing a draft version (a bloody good one, mind), is that it’s a genre that soaks up a whole pile of evidence, while rather hanging rhetorical flourishes out to dry.

Or, to put it another way, this exercise will expose universities that try to hide behind rhetoric. We all say the same stuff anyway – student-first, research-led, challenge and stretch, etc. – so there’s no pretending that one place has uncovered the holy grail of excellence. And I have it on good authority that the TEF panel will be employing state-of-the-art bullshit-detection systems.

So, to take one example, a university might say ‘we’re committed to supporting widening-participation students’ in fifty-eight different ways, but without some evidence of actions, those claims are going to look worse than threadbare. What the university needs to be able to say is: ‘we’re so committed to supporting widening-participation students that we’ve employed people to support them’. That kind of evidence could, let’s say, be worth it’s weight in gold.

I expect, therefore, that the experience of filling those fifteen pages has demonstrated, to managers across the country, the importance of credible commitments and actions. Once the results are announced and all relevant documents are published, I also expect that this message will be reinforced by the experience of us all reading ‘gold’ statements. That will in turn foster a spirit of emulation and competition.

 

I don’t particularly like the TEF: it’s onerous, it’s not especially necessary, and it’s set to produce some perverse results. But I’m coming around to its mind-focusing powers. We’re all accustomed to the discourse of strategy-documents and marketing, but the provider submission will prompt senior managers, on a regular basis, to reflect upon what really has been done to improve the student experience. Maybe ‘working-class officers’ will remain a niche career-path, but I’d predict increased commitments across the country to resources, education enhancement and – maybe above all else – student-support. That can’t be a bad thing.

Prizes for the elite: gold medals and mission groups

The latest rumour from the May government’s efforts to square the circle on international students takes an unexpected turn. It is a proposal to deal with the long-running bickering over post-study employment rights of international students.

According to The Sunday Times, ‘the government is considering allowing overseas students who attend one of the 24 universities in the elite Russell Group to work in Britain after graduating. Those at other universities might be required to return home.’ This would allow the government to claim it is doing something to support universities and international relationships, while still maintaining control of immigration.

According to The Sunday Times, this was all foreshadowed by Amber Rudd’s Conservative party conference speech, in which she referred to possible preferential treatment for ‘our best universities’. At which point we all thought, ‘Ah yes, the Russell Group.’ Or maybe we thought nothing of the sort.

Actually, many of us probably thought this was code for the judgements we’re expecting from the TEF. A strong TEF result – ‘gold’, according to the current proposal – might quite logically lead to preferential treatments of this kind. The possibility that the Russell Group might be used instead as a proxy definition of the UK’s ‘best’ universities poses some interesting questions.

 

Do they know what they’re doing?

This is the year when the irrational has become the norm. But it’s also – to add a little perspective – been a year of lots of talk and not a lot of resolution. It’s a year when we’ve learned to be prepared for anything, but not really to believe it until we see it.

But one common thread in the Conservatives’ stance towards universities has been a determination to draw lines through the sector. The wisdom of this is an open question. It’s not hard to find countries that give preferential treatment, usually in terms of research funding, to selected groups of universities. Concentration of resources in this way can support elite groups to compete internationally in research, while a mass education system is sustained in parallel. The counter-argument would be that the UK’s strength has been built on the back of a broad and fiercely competitive structure, that has produced excellence, in different shapes and sizes, across the board.

If, however, we take the desire to draw lines of demarcation as a given, what would it mean to do so by accepting the Russell Group as the privileged elite? The Russell Group is a mission group that maintains control over its own membership. Traditionally entry was determined overwhelmingly by volume of grant income; however, the defining supremacy of that criterion was frayed somewhat by the last round of entry in 2012, which arguably signalled a commitment to a more rounded model of excellence. It’s possible to join the Russell Group, but almost impossible to be asked to leave.

In recent years the Russell Group has done one thing astonishingly well: branding itself as the ‘elite’ group of UK universities. This is understated in its public mission statement, but a huge part of its public relations and marketing operation. At government level, among busy people, often with only superficial knowledge of their briefs, the idea of an accepted elite group must feel comfortingly easy to grasp.

Russell Group universities have never (to my knowledge, anyway) been given special favours by government, simply by virtue of their membership of the mission group. Hence the elaborate mechanisms of the REF, TEF and QAA, which apply equally to all. The present suggestion therefore feels, potentially, like a pivotal moment.

 

What about the TEF?

One question I’ve been asking since the TEF was first proposed goes roughly along the lines: what happens when people at the highest levels of government realize that some of the universities most likely to lose from the exercise are some of the country’s biggest higher-education brands?

If we follow the proposed logic of the TEF, some universities currently low on the national league tables, though doing very well on teaching metrics, could end up with the right to charge higher fees than some that are very high in the international tables. Existing education metrics, such as the NSS, certainly point in this direction. At that point, it seems to me, the commitment of policy-makers to objective tests of teaching quality, with rewards to match, will be sorely tested.

I wonder whether this scenario is only just dawning on people at the top of government: people conditioned into thinking that there is an ‘elite’ group of universities, bound to rise to the top on any measure of performance. While a decision on work-study visas is only one issue – something, indeed, that may seem peripheral to many people in the sector – it would be hard not to see it as something of a test-case. If Russell Group universities were to get special treatment on this matter – sidelining, in the process, the objective measures of the TEF – what next?

Of course, some might argue that the TEF was never going to do what it claimed anyway. It will not be a reliable measure of teaching quality. Maybe, but that feels to me like a different argument. It’s one thing to debate whether teaching quality can be measured; it would be another thing altogether to abandon objective measures in favour of ‘distinction by mission group’.

 

This particular suggestion, about work-study visas, feels to me a bit like a political advisor flying a kite. It’s not been picked up by other media outlets, and I’m not betting on it happening. But it’s an interesting moment nonetheless: one that indicates the power and potential of elitism in a sector that has been built on more egalitarian foundations, but is creaking under the political and financial strains of the moment.

‘Fee rises’: a comment on the power of words

David Morris’s recent blog-post for Wonkhe, ‘TEF and Tuition Fees: myths and reality’, dares to state the blindingly obvious: that the White Paper’s ‘rises’ in higher education tuition fees are barely rises at all. Indeed, if we take the White Paper at its word, it is actually proposing a mechanism to prevent some – maybe even most – universities from increasing fees in line with inflation. So that’s leaving many universities with long-term, real-terms cuts in fees.

For all their boldness in implementing change, it seems to me the Tories have been hopeless at controlling the message. Let’s start with ‘fees’. These are an odd kind of fee, levied via taxation and capped for those who never become higher earners. Indeed it seems remarkable that Labour has managed to market its ‘graduate tax’ as radically different, when its effect would be roughly similar for most graduates. The Australian scheme, which dates back to the 1980s, was originally titled the ‘Higher Education Contribution Scheme’; although admittedly this has since been dropped, perhaps in acknowledgement of its slippery PR-speak. Maybe, that is, ‘fees’ seems closer to the mark than ‘contribution’; however, it’s subtly misleading, and therefore unhelpful.

And then ‘rises’. Here’s a question: if the Coalition government in 2010 had bitten the bullet on indexation of fees to inflation, would we even have a White Paper in 2016? And would we be getting a Teaching Excellence Framework, leading as it will to convoluted and fine-grained distinctions in levels of fees, from 2017? I think not. It seems to me that an edifice of quality demarcation has been created – and, worse still, a myth of poor-quality education has been propagated – in order to justify what should have been built into the system from the outset.

So this, I’d suggest, is what happens when a government loses control of the terms of debate. ‘Fee rises’ require, as justification, elaborate structures of regulation. The costs of erecting these structures, and of universities bending this way and that to satisfy them, will likely outweigh the extra income. But the power of language drives us not only into this expense, but ultimately into the nonsense – for those universities that don’t attain the highest TEF grades – of ‘rises’ that frankly won’t be rises at all.

None of this is to argue that fees do not feel like fees to those paying them. Nor is it to argue that the current system is either fair or ideal. It is manifestly neither of those things. My point is merely that the choice of words, and the toxic politics accreting over years to those words, is placing us in nonsensical positions.

I’m going to write more about the TEF itself once I get some time to read the ‘technical consultation’ document. As a mechanism for distinguishing between universities that are good, and not so good, at educating their students, I think it can work well enough (although The Guardian league tables do a fair enough job already at no cost to the sector). But as a vehicle for distributing resources it is surely flawed. It’s overly complicated, likely at once to confuse applicants and downgrade hard-won reputations. Moreover, it fundamentally departs from the otherwise rigorous student-centred logic of the White Paper, putting the university’s interest in raising revenue ahead of the student’s interest in minimizing debt.

So I’ll end with a prediction: I don’t think the proposed equation of TEF gradings and ‘fee increases’ will last, if indeed this proposal is ever implemented in the first place. The TEF will survive and ‘fee rises’ will happen, but they will be decoupled. Someone in BIS, one day, will wake up and acknowledge that what’s currently proposed is mad. Surely they will.

The genius of the Green Paper

Fulfilling Our Potential: Teaching Excellence, Social Mobility and Student Choice, the government’s Green Paper on higher education, will attract a lot of comment. That’s part of the point of a green paper, after all. Yet, for all its flaws, this is not a stupid document; if one peels back the shortcomings, what’s left is a core of canny genius.

This document covers a lot of ground. In the longer term, the chapters about market flexibility may be the most significant. There are also the major structural changes: the end of HEFCE, the birth of the Office of Students, an apparent reprieve for the QAA. There’s an impressive emphasis on widening participation, a frontal assault on the degree classification system, even a section on student finance and sharia law. And the final chapter on reseearch funding is worth a look.

But I’ll leave some of that for another day. In this post I’ll focus solely on the Green Paper’s proposals for a Teaching Excellence Framework.

What was the problem?

The Green Paper’s central premise is that too many universities are prioritizing research at the expense of teaching.

That’s debatable; it’s not what I see in my patch, nor does the Green Paper do much more than assert the position. In fact there’s rather too little evidence and too much righteous indignation. It’s saying to university leaders, in effect: we sorted out the income stream for you, and now you’re spending the money on research, and getting distracted by global league tables. But if we accept the premise – just for a minute, just for the sake of argument –  we can appreciate the problem the Green Paper is addressing.

The challenge, in short, is to engineer a system that will convince university managers to redirect institutional resources from research to teaching, at a time when the state has no more money to throw at the sector.

The pot of reputational gold

The REF earns money for universities, in the form of ‘quality-related’ payments. For the TEF, the Green Paper shamelessly exploits the undone business from the last round of reform. The failure to index £9000 fees to inflation was a weak act of political neglect, and had to be fixed one way or another.

So this proposal effects a fix, with strings attached. The arrangements for next year are transitional – and, given the neglible rate of inflation, barely consequential – so let’s focus on the longer term. As I read the proposals (and cf. here the visualization at wonkhe.com), the system will look something like this.

  • Any university that earns the QAA kitemark – as almost all established universities do – will be permitted to increase its maximum fees, in any year, by a percentage of the rate of inflation. That percentage remains to be determined; I’d bet it will be at least 50 per cent.
  • In addition, once every five years or so each university will have the opportunity to apply for recognition for higher levels of teaching quality, which will earn the right to increase fees up to the full rate of inflation. This will be a tougher, optional test.

So here’s the clever bit. The financial benefit of applying for the higher levels of recogntion will not warrant the considerable expense, in terms of buureacracy and changes in institutional practices. Yet, while there will be a temptation to opt out, settling for the Level 1 kitemark, the Green Paper is knowingly leveraging ‘reputational advantage’ for all it’s worth.

And in this context, how many vice-chancellors will be prepared to opt out? What sign would that send to stakeholders? To prospective students? I’d call this ‘Johnson’s choice’.

The dimensions of quality

What constitutes success? Some of the measures are expected: student satisfaction, graduate employment rates, retention rates. That’s all fine; these measures command a level of confidence across the sector.

But there’s more: rather a lot more. Much remains to be determined, but there’s some interesting discussion in the Green Paper about learning gain, and also a sensible recognition of associated widening participation implications. Students from lower-performing schools will, on average, gain on those from higher-perfoming schools, if A-Level results are set against degree results. That, by the way, is a courageous thing for a Conservative government to admit.

There’s also an acknowledgement that contact hours can be deceptive; group sizes also matter. And there’s a welcome concern with who’s doing the teaching. The Green Paper targets the prevailing disparity of esteem between outstanding researchers and outstanding teachers. And it asserts that lecturers on full-time contracts, properly trained, will be more effective than casuals. Indeed.

Perhaps most intriguingly, I note the recurrence of a couple of my favourite words: ‘engagement’ and ‘enhancement’. It won’t be easy to assess all this stuff, but the intention is welcome.

The nightmare scenario

Could it all go wrong? The White Paper’s gamble, as I see it, is that a critical mass of established universities will seek the higher levels of TEF recognition, and that standards of education will rise as a result.

But if a significant number of powerful universities opt-out – reasoning that concentrating on research income, international reputation and international students makes more sense than scrabbling after higher-level TEF recognition – we could get some perverse outcomes. Let’s say, for example, that the LSE (second from bottom in the 2015 NSS) decides not to bother, while Liverpool Hope (seventh from top) earns the top TEF grade. That would leave the LSE with lower fees than Liverpool Hope, and the sector in danger of collapsing into a credibility gap.

In this context, the Russell Group’s response to the Green Paper must surely carry some weight.

‘Reducing regulation’: really?

There’s been a lot of talk in recent years about universities winning independence from government regulation. The Green Paper, whatever noises it makes to the contrary, calls a halt to all of that. It may not fully succeed in putting students at the heart of the system – the goal it inherits from its forebears – but it definitely puts the state right back into the mix.

The TEF: measuring excellence or adequacy?

On the whole, I’m a TEF-enthusiast. I think there’s good and bad teaching, and I think we should be celebrating the fact that in the UK there’s an awful lot more of the former than the latter, despite some public perceptions to the contrary. In the process, we should be exposing the exceptions. I also tend towards the view that it’s not beyond the wit of academics to create a credible basket of metrics that would serve the purpose.

In principle, I also agree with the current government that in order to have any clout, TEF results need to have financial consequences. But it’s the practice of this that might become problematic. The proposal, to the extent that it’s been articulated, is that ‘success’ in the TEF would enable a university to increase fees in line with inflation. That would send a strong message about the value of teaching, and in the process would solve the political problem of a fee structure that nobody thought to future-proof.

But might there be a problem all the same? What I want to do here is tease out some of the potential implications for the TEF of the financial consequences to which it appears that it will be yoked. These will be very different in nature to those we have come to accept from the Research Excellence Framework, and they will shape the impact and value of the TEF. Although there’s still plenty of time to address the possible pitfalls, I’m not yet convinced that this will be in the best interests of the TEF itself, to the extent that we want it to promote excellence.

Some of us may be unfortunate enough to remember the summer of 2015 as the summer of fevered REF noise. (Whereas I, perhaps even more unfortunately, will forever have four neat scars on my foot to remember it by – but that’s another story) Anyhow, at a moment when we should all be taking a break, here are my three TEF-questions (which, on reflection, are arguably all versions of the same question).

  1. Will it measure excellence or determine adequacy?

How important will the ‘E’ be in ‘TEF’? With the REF, we’ve become accustomed to finely-grained distinctions in quality assessments. We’ve maybe even got used to the fact that there are at least six ways to end up on top.

But the TEF now looks like it’s being set up – simply thinking here about the nature of the incentive – as a pass/fail test. If a university is ‘excellent’, it will be allowed to increase fees; if it’s not, it won’t. Maybe there will be finer judgements made. Maybe there will even be league tables: we’re pretty good at them in the UK, after all. In practice, though, I fear the model may lack incentives for a university to really commit itself to excellence.

So one question for me is whether the TEF can still be constructed so as to recognize and reward excellence. I can see the political logic of backing away from this goal, and we can all appreciate the challenges of distinguishing, say, between two-star and three-star teaching. But it seems to me, nonetheless, rather a noble goal.

  1. How excellent is excellent?

Any test requires a pass-mark. Some of the initial rhetoric surrounding the TEF suggests that the pass-mark – the level required by a university in order to be allowed to increase fees – might be quite high. This would concentrate minds and shift priorities across the sector; however, is it really very likely?

For, one way of interpreting the announced link between the TEF and fees is that the government wants to get more money into higher education, and is looking for a way of justifying doing so. Following this line of interpretation, one might assume that most universities are rather expected to pass. Surely the political consequences of, say, several Russell Group universities failing the test, would be more than the current government would want to tackle.

Hence my concern, again, that the ‘E’ might gravitate towards an ‘A’: maybe we’re heading towards a Teaching Adequacy Framework.

  1. How deep will it drill?

The beauty of the NSS is that it gives us finely-grained data at discipline level. I’m never too fussed about the difference between, say, an overall satisfaction rate of 90 or 94, despite the fact that this might mean twenty places on a league table. But every year, without fail, the NSS focuses my attention, in credible and specific ways. It did so when I was responsible for educational quality across multiple departments, and was sensitive to outliers; and it does so as head of department, when I’m looking for areas of weakness to target in the coming year.

But will the TEF give us discipline-level results? It seems hard to imagine a model that would allow one discipline within a university to increase fees, while the discipline down the corridor is denied this flexibility. That may achieve a feeling of justice from the outside; however, there’s not much sense in it as far as universities are concerned. But if this doesn’t happen, will pockets of under-performance be exposed at all? It’s very unusual to get a crap university, but much more common to find crap programmes within an adequate university. A successful TEF would identify them.

So, once again, I can see a tension here: between the need to construct a system that produces the neat pass-fail institution-level results, and the desire to create a model that genuinely recognizes different levels of performance.

Conclusion

I think a TEF can be done, but I’m not convinced it can genuinely drive forward excellence and drive out weaknesses if the financial incentives are established in the way that as been foreshadowed. The drive to promote excellence is just not wholly compatible with a mechanism to recognize adequacy. It will help, because it will provide a lever to increase fees, but I wonder whether we may look back on a missed opportunity.

Or maybe there’s still time to address these concerns.