REF2021 update 4 - What is a 3* or a 4* star paper?
11th June 2019
From: Professor Jodi Lindsay
Dear colleagues,
The decision to score a paper as 3* or 4* will be made by the REF2021 assessment panels in 2021. This email update is designed to summarize 1) the guidance from REF2021 about what the panels are looking for, 2) information provided from previous panel members, and 3) tips about publication metric indicators.
In addition, the email also covers
4) A reminder about how many papers SGUL will submit to REF2021
5) How SGUL will choose the papers to submit
6) A brief summary of SGUL’s first mock REF assessment of 3* and 4* papers
Apologies that the email is long!
1) Firstly, the REF2021 guidance states that outputs (papers) will be assessed on originality, significance and rigor. Specifically -
Originality will be understood as the extent to which the output makes an important and innovative contribution to understanding and knowledge in the field. Research outputs that demonstrate originality may do one or more of the following: produce and interpret new empirical findings or new material; engage with new and/or complex problems; develop innovative research methods, methodologies and analytical techniques; show imaginative and creative scope; provide new arguments and/or new forms of expression, formal innovations, interpretations and/or insights; collect and engage with novel types of data; and/or advance theory or the analysis of doctrine, policy or practice, and new forms of expression.
Significance will be understood as the extent to which the work has influenced, or has the capacity to influence, knowledge and scholarly thought, or the development and understanding of policy and/or practice.
Rigour will be understood as the extent to which the work demonstrates intellectual coherence and integrity, and adopts robust and appropriate concepts, analyses, sources, theories and/or methodologies.
The sub-panels will assess outputs on the above criteria and will apply the generic definitions of the starred quality levels: 4* is world leading in terms of originality, significance and rigour; 3* is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence.
SGUL will be submitting to Units of Assessment (UoA) 1 (Clinical Sciences) and UoA2 (Population Health). These sub-panels will look for evidence of 3* or 4* quality using some of the following indicators:
• scientific rigour and excellence, with regard to design, method, execution and analysis
• significant addition to knowledge and to the conceptual framework of the field
• actual significance of the research
• the scale, challenge and logistical difficulty posed by the research
• the logical coherence of argument
• contribution to theory-building
• significance of work to advance knowledge, skills, understanding and scholarship in theory, practice, education, management and/or policy. Applicability and significance to the relevant service users and research users
• potential applicability for policy in, for example, health, healthcare, public health, food security, animal health or welfare. (Source - https://www.ref.ac.uk/media/1084/ref-2019_02-panel-criteria-and-working-methods.pdf paras 190-201)
2). Information from REF2014: attached is a copy of some personal advice kindly shared by Prof Neil Gow, currently Deputy Vice-Chancellor (Research and Impact) at University of Exeter, which I presented last
summer in my oral presentations. Neil was on the UoA5 (Biological sciences) panel for REF2014, on the panel advising the new REF2021 criteria and will be on the REF2021 UoA5 panel.
3). Publication metric indicators: following an analysis of REF2014 results to determine if REF2021 could be replaced by metric indicators (https://responsiblemetrics.org/the-metric-tide/), it was concluded that metric indicators alone were not sufficient to capture research quality. Interestingly, for UoA1 and UoA2 3* or 4* assignment by Panels correlated well with citation counts (adjusted for subject and year of publication). REF2021 will provide citation counts and context of subject and year to the UoA1 and UoA2 Panels, and Panels are instructed not to use any other metrics (in particular, journal impact factors). ‘Citations will be used as appropriate as a potential indicator of academic significance to inform the assessment of output quality’ (para 274-7). Citation counts will be provided by Clarivate/Web of Science, and SGUL subscribes to this service (http://libguides.sgul.ac.uk/az.php?a=w0 ). In late summer, we will run a session and provide some resources on publication metrics.
4). The REF rules require that every returned staff member will have their best paper submitted, and the maximum number of papers we can submit attributable to an individual staff member is 5. We will submit a total of 2.5 papers per full-time equivalent (FTE) staff for each UoA (https://www.ref.ac.uk/media/1092/ref-2019_01-guidance-on-submissions.pdf, paras 2017-210). So for example, if we have 100 FTE staff in UoA1 on the census date of July 31 2020, we will submit 250 papers to UoA1, including at least one per staff member. We can also include papers from staff employed between 2014–2020 who have since left.
5). In SGUL’s Code of Practice (submitted June 5 2019) we have proposed how we will identify which papers we submit. Note that this is a little different from us predicting which will be scored 4* and which will be scored 3*.
1. All outputs considered for REF to be collected in CRIS (Lead: Library)
2. Staff will be asked to put forward their favoured top 6 papers (optional, in CRIS)
3. Eligible outputs identified in CRIS (Lead: JRES)
4. Metrics used to initially rank outputs (Lead: JRES)
5. Internal panels appointed for each UoA to read and score papers individually. All staff will be eligible to apply. Approximately eight to twelve will be chosen for experience (e.g. on research panels, journal editor). They will not necessarily be REF Steering Executive members. Equality, diversity and inclusion of panel members will be considered. Training will be provided and individual scoring collected electronically.
6. A selection of papers will be scored by external assessors.
7. The strongest paper for each returned academic staff member will be selected for submission.
8. The remaining papers will be ranked, and the top papers chosen until cut-off of FTE x 2.5 reached, removing those papers where all eligible SGUL authors on the paper have 5 others ranked more highly.
9. Final decisions confirmed by REF Steering Executive and approved by the Research Committee.
6) How does our data look at the moment? SGUL will have some very nice 4* papers to submit and a surplus of 3* papers, which is a solid position to be in. Obviously, more 4* papers will increase our income and our reputational scores. The REF steering committee and the new Internal panels will be tasked with identifying the cut-off point of which papers are submitted and which we think are a little less strong. We will be calling for volunteers for those Internal panels to help with assessing papers shortly.
Good luck with your 4* papers!!
Best wishes
Jodi
Academic Lead for REF2021