If you are a registered HEi-know user, please log in to continue.
You must be a registered HEi-know user to access Briefing Reports, stories and other information and services. Please click on the link below to find out more about HEi-know.
Interventionism is suddenly all the rage with the Westminster Conservative government, and higher education is feeling the impact as new policies and legislation are brought to bear on the sector, writes Johnny Rich, Chief Executive of Push and of the Engineering Professors’ Council.
Mike Boxall, an independent researcher and consultant on higher education policies and strategies, and a senior adviser to PA Consulting, considers the emerging post-COVID world and its implications for the future of universities. His blog is based on a paper published recently by PA Consulting, and co-authored with its HE lead, Ian Matthias.
With the deadline for HE sector responses to the government’s consultation on plans for the subject-level Teaching Excellence Framework rapidly approaching, Professor Jon Scott, a TEF panel member and Pro Vice-Chancellor (student experience) at the University of Leicester outlines the proposed next steps and options in the evolution of the TEF.
As everyone in the HE sector will be aware, the Teaching Excellence Framework is evolving rapidly. The provider level process that was first undertaken as TEF Year 2 in 2016-17 has developed in its 2017-18 iteration with the halving of the weighting of the National Student Survey (NSS) metrics and the inclusion of Longitudinal Education Outcomes (LEO) and grade inflation metrics. This change reflects a greater emphasis on output versus input measures: the weighting in the metrics attached to the students’ perceptions of the quality of teaching, of assessment and feedback and of academic support, has been halved while the inclusion of the LEO data adds in sustained employment and salary data, alongside the existing Destination of Leavers from Higher Education (DLHE) data, but as yet lacks critical benchmarking factors such as geographical location.
Alongside the provider level TEF assessments, 2017-18 has also seen the first round of pilots for the Subject Level TEF, with the object being for this exercise to be fully implemented in 2019-20 (TEF Year 5 for those who like to count). Some 50 HE providers, across the full range of provision, participated in the pilot exercise, making submissions against one or both of the two models being tested. In line with the progressive marketisation of higher education, one of the main drivers for the introduction of the Subject Level TEF is the provision of more effective information, advice and guidance for students backed, among others, by the Competition and Markets Authority, which recommended that this should be implemented as soon as possible.
The DfE is therefore undertaking three parallel exercises: evaluation of the subject pilots, a consultation exercise involving the HE providers, and student research. The student research will focus on the perceived utility of the classification systems for applicants and what factors related to ‘teaching quality’ are seen by students as being important when making their choice of provider. These factors are likely to include aspects such as employability prospects, contact time, quality of teaching staff and the quality of learning environment.
In addition, in 2019 there will still be the independent review of the TEF, as set out in the Act.
Subject Level TEF will be the basis of the TEF from 2019-20 onwards, with each provider being rated at the provider level and at the level of each of the subjects that is taught. The identification of the subjects is based on the 35 ‘subjects’ derived from level two of the Common Aggregation Hierarchy (CAH2). The CAH is the system that will replace JACS coding for UCAS applications and HESA reporting.
In Model B of the pilot, ‘bottom up’ evaluation, the 35 subjects are aggregated into seven disciplinary areas. The provider drafts a subject group submission against the metrics for each of the seven areas along with a provider level submission which is focused on three criteria: valuing teaching (TQ2), learning resources (LE1) and positive outcomes for all (SO3). The ratings are then awarded at subject level and the provider-level rating is informed by the aggregated assessments of the subject ratings, the provider metrics and the provider submission.
In Model A, ‘by exception’, each subject is rated and referenced against the provider-level rating. Where the metrics give similar outcomes, the subjects are awarded the same rating as the provider. Where the subject-level metrics differ from the provider level, the provider drafts a submission for each of those subjects and is also able to select a small number of additional subjects. The provider also drafts a full provider-level submission addressing all 10 TEF criteria.
The Consultation will run until the 21st May and explores the views of the sector regarding the implementation of the Subject TEF and the inclusion of the Teaching Intensity metrics. Full details of the consultation can be found at on the DfE website, along with the supporting technical documentation.
There are 16 questions posed in all. The core questions focus on whether the exercise should be a form of Model A or B (or an alternative approach, to be specified), how the subject-level ratings should inform the provider-level rating, whether the CAH2 provides the appropriate level of subject identification and whether the Subject TEF should retain the features and TEF criteria used in the current framework.
The duration of the rating is discussed, with two options being considered: the first option is based on an annual TEF assessment exercise with the provider ratings being valid for up to five years. The annual exercise would enable new providers to enter the exercise without having to wait several years for the next TEF round. There would also be a minimum re-application period, such that a provider could not re-apply for a period of three years following a submission irrespective of their rating.
The second option is of increased duration to account for the additional assessment time required for Subject level TEF and the cost for the sector. In this case, the assessment exercise would run every other year, the award duration would be up to six years and the minimum re-application period would be four years. Participation in the Pilot exercise has indicated to institutions that there is a significant cost associated with the work and there will clearly also be a significant operational cost in managing the assessment exercises, but these costs have to be weighed against ensuring that the information being provided is current: when a student is choosing a course to start the following year based on a published rating that is six years old, and which is based on metrics from preceding years’ data, it begs the question how valid that rating is as a guide to the student’s choice.
Another challenge for Subject TEF which is explored in the consultation is that of non-reportable metrics. A feature of the TEF Year 3 round of provider-level applications is that it includes a greater proportion of small providers, for example alternative providers and HE in FE provision. For these smaller providers there is an increased probability that they will not have sufficient data for the metrics to be statistically valid. In the case of subject-level TEF, the sub-division of the student populations across the 35 subjects is likely to result in many providers having some subjects where the core metrics are non-reportable. This will be exacerbated when considering the split metrics to identify different student groupings.
In the consultation, the proposal for addressing non-reportable core metrics is based on a combination of three factors: a lower threshold for the metrics, whereby at least two but not all six core metrics have to be reportable; using the metrics based on the seven subject groupings from Model B alongside any reportable metrics or using the provider-level metrics alongside any reportable metrics for that subject. The DfE does not propose consulting on other options such as increasing the weighting of the submission or rely on other data supplied by the provider.
Teaching Intensity: the final set of specific questions in the consultation focus on the proposed teaching intensity metric. It is noted that “there is strong evidence that teaching intensity, including factors such as contact hours and class size, matter to students’ perceptions of their studies” and that the amount of teaching delivered within a given discipline can vary markedly between providers.
In the initial pilot exercise, the intensity data was collected based on the Gross Teaching Quotient (GTQ) and also a survey of students’ perceptions of the amount of teaching they received. The GTQ was based on reported contact hours weighted by the ratio of students to teachers in the class, along with measures of additional activities such as field-work, work-based learning and on-line teaching. In addition, there was a survey of the student body regarding their perceptions of the quantity of teaching they received. Although it is based on class size, the GTQ does not take any account of the mode of delivery. Thus a non-interactive lecture delivered by a member of staff to a group of 5 students would be rated six-times better than a highly interactive session with 30 students.
The consultation includes possible modifications to the GTQ, for example by weighting the contact time based on both the number of students and the seniority and/or qualification of the member of staff with that being taken as a proxy for the quality of teaching. Another option proposed is along the lines of the Key Information Set (KIS) that was previously published by providers, indicating the expected time allocations for different activities within a programme. Further possible measures are based on engagement data derived from learner analytics approaches or measures of the numbers of hours staff are contracted to teach. It is recognised that all of these measures present specific challenges in terms of collection and interpretation of the data, especially if they are to provide a valid reflection the quality of the student learning experience.
The timelines for all these activities are tight: the plan is that the learning outcomes from the subject pilots, the consultation and the student research will inform a governmental response scheduled for autumn 2018, which in turn will form part of the evidence base for the Statutory Independent Review to be published in the summer of 2019. This, along with the outcomes of the second stage Subject Level TEF will inform the full implementation of the Subject Level TEF for TEF year 5 in 2019-20.
This article was first published by HEi-know on 14 May 2018
© 2013 Media FHE, all rights reserved