Standards as a social contract in curriculum-based qualifications

Key Question Being Explored in This Insight:

How do stakeholder perceptions of qualification standards in Scotland, particularly during the COVID-19 pandemic, reflect tensions between different assessment paradigms?

View the page of: Jo-Anne Baird

Institution: University of Oxford

Explore more insights in:

Education Literacy, Language and Communication Professional

Media Literacy Toolkit

Faculti's Media Literacy Toolkit helps viewers critically engage with academic insights by analyzing the research context, identifying perspectives, and encouraging thoughtful evaluation.

Critical Questions to Consider ▼

What assumptions does the research make?
Are there alternative perspectives not explored?
What are the limitations of the research method?

Bias and Perspective Awareness ▼

This research comes from University of Oxford . Reflect on how the institution's academic focus and research partnerships may shape the questions being explored.

Further Reading ▼

Read the Study

How do stakeholder perceptions of qualification standards in Scotland, particularly during the COVID-19 pandemic, reflect tensions between different assessment paradigms?

Image courtesy of interviewee. January 17, 2025

Log-in or Sign-up to Faculti
Currently viewing this subject insight as a guest. You have insight(s) remaining for this month. Login to view 8000+ figures on the platform.

Generating Workspace...

Export as PDF

This video discusses research conducted in Scotland during the pandemic focused on stakeholders' views regarding exam standards. The study aimed to bridge the gap between theoretical concepts and public understanding, assessing views from learners, parents, teachers, and employers.

Key findings revealed that the majority (60%) viewed standards as reflecting student performance in assessments based on predetermined criteria, while 40% thought they represented a stable underlying construct, reflecting individual ability. A smaller group believed standards were based on cohort referencing, where grades are relative to past performances.

The research highlighted that public understanding of assessment is low, and even teachers often lack assessment literacy due to centralized exam systems. These misunderstandings lead to public discussions that may not align with actual practices in assessment.

This video also discusses the cultural specificity of national examination systems, illustrating how different countries have varying practices and beliefs about assessments. For example, the length and security practices surrounding exams in Malaysia, the teacher grading systems in the UK during the pandemic, and oral exams in Italy are contrasted.

The pandemic challenged existing norms, leading to a public outcry over the fairness of statistical moderations used in grade assessments, prompting a reversion to teacher-assessed grades amidst concerns of socioeconomic bias. Despite the challenges, trust remained largely intact in exam boards, emphasizing that confidence in the system is essential for its value.

Overall, the discussion underscores the complexities of assessment practices, the essential role of trust, and the impact of social dynamics on the perception of educational standards.

Language:

Export as PDF

Export as PDF
Here are five key points from the video regarding the research conducted in Scotland on exam standards during the pandemic:

1. **Stakeholder Perspectives on Standards**: The research aimed to understand how various stakeholders (learners, parents, teachers, employers) conceptualized exam standards during the pandemic. It revealed that the predominant view among participants was that standards reflect student performance on assessments per established grading criteria, while a notable minority saw standards as representative of stable individual traits like ability or intelligence.

2. **Low Public Understanding of Assessment**: The study highlighted a significant lack of public understanding and literacy regarding assessment and qualification standards. This gap in understanding is not just present among the general public but is also reflected among teachers, indicating a widespread challenge that educational authorities need to address globally.

3. **Cultural Convideo of Examinations**: The video emphasized that national examinations and assessment methods are culturally specific and influenced by various social and historical factors. Different countries employ distinct practices and criteria, leading to varying perceptions and acceptability of assessment standards across global educational convideos.

4. **Impact of the Pandemic on Assessment Systems**: The pandemic disrupted traditional assessment practices, leading to increased reliance on online data collection methods and a shift from statistical moderation of grades to teacher-assessed grades in the UK. This transition was fueled by public outcry over perceived unfairness in statistical moderation, which raised questions about socioeconomic biases impacting student results.

5. **Trust and Public Discourse on Meritocracy in Assessments**: Despite the turmoil and changes brought about by the pandemic, a notable level of trust in examination boards remained. The video described the political implications of assessing standards, especially concerning the meritocratic nature of assessments, highlighting the broader societal discourse on fairness and equity in educational outcomes that was significantly scrutinized during this time.

Export as PDF
Here are some key quotes pulled from the video that highlight important points concerning exam standards, public understanding, and the impact of the pandemic on assessment practices:

1. **On the purpose of research during the pandemic:**
- "During the pandemic, we were interested in stakeholders' views... we collected data on our theoretical conceptualizations of standards."

2. **On public understanding of assessment:**
- "Public understanding of assessment is actually quite low... centralized in most countries, teachers don't have a lot of assessment literacy either."

3. **On conceptualizations of standards:**
- "We found was that in Scotland... the most popular view of standards was that they represent the performances that students have on the assessment on the examination."

4. **On cultural differences in national examinations:**
- "What I really want to get across is that national examinations are culturally bound to the convideo in which they're conducted."

5. **On the impact of the pandemic on the established understanding of assessments:**
- "The pandemic punctured the social settlement that we had around standards that were being raised."

6. **On the political implications of assessment:**
- "This social contract then is much broader than the idea of using techniques, scientific techniques and incorporates wider social practices and views about what standards mean."

7. **On public discourse during the pandemic:**
- "The tensions around whether they have ever been reflective of a meritocratic system have also come to the fore far more so during the pandemic."

8. **On the response to the pandemic-related changes in assessment:**
- "In Scotland and England and Northern Ireland and Wales... political change on the day that results were released to move away from the statistically moderated grades that had been issued, back to the teacher, assessed grades."

9. **On trust in examination boards post-pandemic:**
- "There was a surprising maintenance of trust in the exam boards in the face of this."

10. **On examination popularity:**
- "A bizarre aspect of the pandemic was that examinations have never been so popular since they were removed."

These quotes capture the essence of the speaker's insights on educational standards, the complexities of stakeholder perceptions, and the interplay of societal factors in shaping assessment practices, especially during the upheaval of the pandemic.

X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:90000

1
00:00:00.265 --> 00:00:01.605
I'm gonna talk specifically

2
00:00:01.625 --> 00:00:04.565
to some research we did in Scotland,

3
00:00:04.585 --> 00:00:06.205
and the during the pandemic.

4
00:00:06.505 --> 00:00:10.085
And the reason for that is that most of the research on

5
00:00:10.355 --> 00:00:12.165
what exam standards mean,

6
00:00:12.225 --> 00:00:14.965
or high stakes qualification standards mean

7
00:00:15.625 --> 00:00:16.845
is actually theoretical.

8
00:00:17.745 --> 00:00:18.765
So over the years,

9
00:00:18.895 --> 00:00:22.485
we've been developing ideas about the different kinds of

10
00:00:23.385 --> 00:00:26.485
conceptualizations people have, about what standards mean.

11
00:00:26.625 --> 00:00:28.445
So during the pandemic,

12
00:00:28.515 --> 00:00:30.485
this was really my first opportunity

13
00:00:30.665 --> 00:00:33.085
to collect data directly.

14
00:00:33.205 --> 00:00:35.605
I had been analyzing the kind of public debates

15
00:00:35.605 --> 00:00:37.005
that there have been and so on,

16
00:00:37.505 --> 00:00:39.805
and looking at how people conceptualized them.

17
00:00:40.585 --> 00:00:42.165
But during the pandemic in Scotland,

18
00:00:42.945 --> 00:00:45.325
we were interested in stakeholders views.

19
00:00:45.465 --> 00:00:48.205
By that I mean, learners, their parents,

20
00:00:49.045 --> 00:00:51.205
teachers, employers, and so on.

21
00:00:51.385 --> 00:00:54.885
So we collected data on our theoretical

22
00:00:55.625 --> 00:00:57.245
conceptualizations of standards.

23
00:00:57.905 --> 00:01:00.325
Now, they weren't gonna understand or use those terms

24
00:01:00.325 --> 00:01:02.685
because we have our own technical jargon.

25
00:01:03.265 --> 00:01:06.165
So what we did was we translated them into statements

26
00:01:06.195 --> 00:01:07.885
that everyone could understand.

27
00:01:08.505 --> 00:01:10.365
We, we piloted that work so

28
00:01:10.365 --> 00:01:11.965
that we knew it was gonna work well.

29
00:01:12.825 --> 00:01:16.685
And what we found was that in Scotland, in the sample

30
00:01:16.715 --> 00:01:19.805
that we had, so we don't know that that's generalizable

31
00:01:19.905 --> 00:01:21.285
to the wider population,

32
00:01:21.985 --> 00:01:25.365
but in Scotland, the most popular view of standards was

33
00:01:25.365 --> 00:01:27.405
that they represent the performances

34
00:01:27.955 --> 00:01:30.125
that students have on the,

35
00:01:30.125 --> 00:01:31.725
assessment on the examination.

36
00:01:32.025 --> 00:01:35.485
So by that they mean conformed with written criteria

37
00:01:35.755 --> 00:01:39.285
that are specified in advance that tell you what you need

38
00:01:39.285 --> 00:01:40.325
to do for each grade.

39
00:01:41.385 --> 00:01:44.445
Now, the second most popular, way

40
00:01:44.445 --> 00:01:46.645
of thinking about standards was

41
00:01:46.645 --> 00:01:48.485
that they actually represent,

42
00:01:48.805 --> 00:01:52.885
a construct which is underlying the performances,

43
00:01:52.925 --> 00:01:55.245
which tells you something more about the individual

44
00:01:55.905 --> 00:01:57.445
and is probably more stable.

45
00:01:58.105 --> 00:02:00.725
So, for example, you might think of this in terms of ability

46
00:02:01.145 --> 00:02:04.845
or intelligence, and about 40% of our sample

47
00:02:05.685 --> 00:02:06.805
actually responded in that way.

48
00:02:06.805 --> 00:02:10.085
It was six 60% for, the first,

49
00:02:10.895 --> 00:02:14.595
but there was also, and these people didn't use standards

50
00:02:14.595 --> 00:02:15.795
terms in just one way.

51
00:02:15.975 --> 00:02:18.835
So people, agreed with more than one of the items,

52
00:02:19.615 --> 00:02:23.195
but about one fifth of the population who responded

53
00:02:23.865 --> 00:02:27.195
also said that they thought standards represented the same

54
00:02:27.245 --> 00:02:29.235
percentage, getting the grades over time.

55
00:02:30.255 --> 00:02:32.595
So we call that, cohort referencing.

56
00:02:32.665 --> 00:02:34.395
Some people call it non referencing,

57
00:02:34.685 --> 00:02:36.435
which is actually a little bit of a mistake.

58
00:02:37.175 --> 00:02:39.435
Um, but they talk about grading on the curve

59
00:02:39.575 --> 00:02:41.195
and under normal distribution.

60
00:02:41.775 --> 00:02:45.235
So these are the, the kinds of, most popular ways

61
00:02:45.235 --> 00:02:47.595
that people talked about standards, though there are others.

62
00:02:47.775 --> 00:02:51.395
So one thing to say is that, public understanding

63
00:02:51.495 --> 00:02:54.155
of assessment is actually quite low.

64
00:02:54.365 --> 00:02:56.395
There are lots of things that are accepted

65
00:02:57.055 --> 00:02:58.595
as the way that they're

66
00:02:58.595 --> 00:03:01.685
Practiced, when in fact it's, it's actually wrong.

67
00:03:02.475 --> 00:03:04.005
Another thing to say is that

68
00:03:04.075 --> 00:03:06.645
what we know from the literature is that this isn't just

69
00:03:06.955 --> 00:03:07.965
amongst the public.

70
00:03:08.635 --> 00:03:11.805
That actually, because high stakes assessments are largely

71
00:03:12.075 --> 00:03:15.565
centralized in most, most countries,

72
00:03:16.045 --> 00:03:18.685
teachers don't have a lot of assessment literacy either.

73
00:03:18.905 --> 00:03:20.965
So this is something that governments

74
00:03:20.965 --> 00:03:23.165
around the world are contending with

75
00:03:23.585 --> 00:03:26.005
and trying to educate teachers more about.

76
00:03:27.245 --> 00:03:30.705
So this means that the public discussions are often at odds

77
00:03:30.705 --> 00:03:32.265
with the, the facts about

78
00:03:32.325 --> 00:03:34.625
how things are actually done to add to this.

79
00:03:35.445 --> 00:03:37.705
Um, understandings

80
00:03:37.705 --> 00:03:40.705
of assessment differ across different countries,

81
00:03:41.125 --> 00:03:42.825
and we have really quite different systems.

82
00:03:43.995 --> 00:03:46.725
This is something that I noticed when I started going

83
00:03:46.725 --> 00:03:50.205
to conferences and hearing people speak about the,

84
00:03:50.355 --> 00:03:51.805
controversies in their systems,

85
00:03:51.805 --> 00:03:55.525
which just wouldn't have been controversial in my setting

86
00:03:56.345 --> 00:03:58.765
or talking about systems that they had

87
00:03:58.765 --> 00:04:02.605
that would've been hugely controversial, in the uk.

88
00:04:03.685 --> 00:04:06.905
So just to give you a flavor of this, for example,

89
00:04:07.045 --> 00:04:10.105
in Malaysia to keep the exams secure,

90
00:04:10.965 --> 00:04:14.025
the examiners are held in a compound

91
00:04:15.165 --> 00:04:16.345
to write the exams.

92
00:04:16.485 --> 00:04:19.025
And then between the writing of them

93
00:04:19.085 --> 00:04:22.265
and the actual setting of the exams, they're held there.

94
00:04:22.445 --> 00:04:23.705
So that's over a month.

95
00:04:24.865 --> 00:04:27.005
And this is, it's been shown on tv.

96
00:04:27.385 --> 00:04:31.685
The examiners going, in a convoy to this compound.

97
00:04:31.805 --> 00:04:34.565
I don't think that would be acceptable practice here in the

98
00:04:34.625 --> 00:04:37.605
uk, but there are other examples like

99
00:04:38.495 --> 00:04:39.915
how the assessments are done.

100
00:04:40.135 --> 00:04:43.155
So use of multiple choice tests with the idea

101
00:04:43.155 --> 00:04:44.795
that there's a single right answer

102
00:04:45.135 --> 00:04:48.755
for a politics exam in China, that's part of the GCO exam,

103
00:04:48.755 --> 00:04:53.695
that that wouldn't be acceptable in lots of countries,

104
00:04:53.835 --> 00:04:56.015
or in gco until very recently,

105
00:04:56.235 --> 00:04:59.855
you needed higher marks in the exam to enter universities.

106
00:04:59.915 --> 00:05:01.735
And the most selective universities,

107
00:05:02.315 --> 00:05:05.455
if you start the exam in some of the rural constituencies,

108
00:05:06.275 --> 00:05:09.735
which again, we wouldn't find, acceptable,

109
00:05:10.915 --> 00:05:13.175
but in Ireland, they have exams

110
00:05:13.175 --> 00:05:14.975
where the pass marks don't change

111
00:05:15.785 --> 00:05:17.415
every year as they do here.

112
00:05:17.915 --> 00:05:21.775
And it would be outrageous there if, people wanted

113
00:05:21.795 --> 00:05:23.095
to change the pass marks.

114
00:05:23.635 --> 00:05:25.815
So to keep that system stable, they actually have

115
00:05:25.815 --> 00:05:29.055
to remark some of the exams to make sure

116
00:05:29.055 --> 00:05:30.735
that the distributions aren't,

117
00:05:31.565 --> 00:05:33.495
very different every year, the distribution

118
00:05:33.495 --> 00:05:34.615
of students getting the grades.

119
00:05:34.715 --> 00:05:37.135
But if you think about even in the us,

120
00:05:37.555 --> 00:05:40.535
the grade point average system that's brought about

121
00:05:40.635 --> 00:05:44.095
by teachers marking students on totally different tasks

122
00:05:44.845 --> 00:05:47.255
with different criteria all across the country.

123
00:05:47.795 --> 00:05:51.615
And those results are used by universities in the US for,

124
00:05:51.875 --> 00:05:55.015
entry requirements and are, and are widely respected.

125
00:05:55.155 --> 00:05:56.535
And just to throw in one more,

126
00:05:56.755 --> 00:06:00.485
In Italy, there's a tradition of oral examinations,

127
00:06:00.975 --> 00:06:04.645
which again, is so different from what we do here in the uk.

128
00:06:05.585 --> 00:06:09.325
So from these examples, what I really want to get across is

129
00:06:09.325 --> 00:06:14.005
that national examinations are culturally bound,

130
00:06:14.345 --> 00:06:17.085
to the context in which they're conducted.

131
00:06:17.625 --> 00:06:18.885
So there are lots of social

132
00:06:19.225 --> 00:06:21.925
and historical drivers about what's acceptable

133
00:06:22.385 --> 00:06:24.605
and what's not in national qualifications,

134
00:06:24.985 --> 00:06:26.925
and that really affects how people think about

135
00:06:26.925 --> 00:06:28.005
what standards are.

136
00:06:28.145 --> 00:06:30.605
And research during the pandemic,

137
00:06:30.605 --> 00:06:32.965
was much more difficult, obviously,

138
00:06:33.465 --> 00:06:37.005
to collect data in the traditional ways that we'd done that.

139
00:06:37.425 --> 00:06:41.125
So lots of data collection moved to online,

140
00:06:41.785 --> 00:06:43.125
methods.

141
00:06:44.025 --> 00:06:46.485
So in Scotland and the research we conducted

142
00:06:46.485 --> 00:06:49.525
during the pandemic, we held focus groups, which meant

143
00:06:49.525 --> 00:06:52.405
that we were bringing people together from across,

144
00:06:52.545 --> 00:06:56.305
across the country, using,

145
00:06:56.705 --> 00:06:58.065
software like Microsoft Teams.

146
00:06:59.165 --> 00:07:00.385
So it's not the case

147
00:07:00.385 --> 00:07:02.065
that this was the first time this was done,

148
00:07:02.085 --> 00:07:04.465
but it was just being done far more frequently.

149
00:07:05.005 --> 00:07:09.185
We also, conducted a survey, with

150
00:07:09.725 --> 00:07:11.345
nearly so altogether,

151
00:07:11.345 --> 00:07:13.945
there were over a thousand people involved in that research.

152
00:07:14.405 --> 00:07:15.785
All of it conducted online.

153
00:07:16.365 --> 00:07:19.945
And really these methods that we used during the pandemic

154
00:07:20.085 --> 00:07:23.945
and are more reliance on technology to do these, these forms

155
00:07:23.965 --> 00:07:26.585
of research have really continued post pandemic.

156
00:07:26.935 --> 00:07:31.025
When we talk about national standards within

157
00:07:31.565 --> 00:07:35.935
the assessment community, we have to, we've had

158
00:07:36.035 --> 00:07:40.015
to over the years, construct rules so

159
00:07:40.015 --> 00:07:41.535
that we can make things fairer.

160
00:07:42.075 --> 00:07:45.295
We have lots of ways in which, the standards are checked

161
00:07:45.515 --> 00:07:48.535
so that ensure the marking is fair

162
00:07:48.595 --> 00:07:51.215
and so on that they question papers reflect the

163
00:07:51.215 --> 00:07:52.655
curriculum and all the rest of it.

164
00:07:53.355 --> 00:07:55.535
Now, to do that, it means that we've had

165
00:07:55.535 --> 00:07:59.055
to construct our own language to describe these processes.

166
00:07:59.515 --> 00:08:01.855
So we have jargon in relation to this.

167
00:08:02.155 --> 00:08:04.135
In other words, we've had to codify the field.

168
00:08:06.225 --> 00:08:10.125
Now that assessment expertise then is quite concentrated in

169
00:08:10.125 --> 00:08:11.445
assessment organizations.

170
00:08:12.585 --> 00:08:16.725
And it means that when we talk to people outside that field,

171
00:08:16.835 --> 00:08:18.405
they may be using different language

172
00:08:18.505 --> 00:08:21.445
and they may not understand, how these terms

173
00:08:21.955 --> 00:08:24.605
reflect quite complex processes,

174
00:08:24.755 --> 00:08:27.165
underline the national assessments.

175
00:08:27.385 --> 00:08:30.485
So a critique of the assessment field is

176
00:08:30.485 --> 00:08:34.445
that we are a techno, a technocracy, that we've set up this

177
00:08:35.085 --> 00:08:37.725
edifice of expertise with our own language

178
00:08:38.265 --> 00:08:40.325
and that we're insiders speaking to ourselves.

179
00:08:41.735 --> 00:08:43.835
And that quite often some of the terms we,

180
00:08:43.935 --> 00:08:47.995
we use don't address public concerns,

181
00:08:48.775 --> 00:08:51.715
in the terms that are, they're being used, publicly.

182
00:08:52.015 --> 00:08:55.605
The thing about the pandemic was this punctured the social

183
00:08:55.775 --> 00:08:58.605
settlement that we had around standards

184
00:08:59.985 --> 00:09:02.205
and made us think about the sorts of concerns

185
00:09:02.205 --> 00:09:05.405
that were being raised, in the terms

186
00:09:05.405 --> 00:09:06.445
that they were being raised.

187
00:09:07.345 --> 00:09:09.525
Now it is, well, for a long time it's been known

188
00:09:09.525 --> 00:09:11.125
that there are always tension

189
00:09:11.985 --> 00:09:15.685
and contested issues around examinations

190
00:09:15.685 --> 00:09:17.805
because they affect people's lives chances

191
00:09:17.825 --> 00:09:19.565
to a large extent.

192
00:09:20.425 --> 00:09:24.085
Um, and that social settlement that we normally have, right,

193
00:09:24.385 --> 00:09:26.365
relies on a system of meritocracy

194
00:09:26.945 --> 00:09:29.845
in which people give up their individual rights

195
00:09:30.705 --> 00:09:33.925
for a wider system that generally works fairly.

196
00:09:34.345 --> 00:09:37.605
So that's the kind of social contracts that I'm,

197
00:09:37.665 --> 00:09:41.825
I'm talking about that there are wider societal benefits

198
00:09:41.825 --> 00:09:45.105
of having a system, that is,

199
00:09:45.615 --> 00:09:47.185
intentionally metastatic.

200
00:09:47.565 --> 00:09:51.385
So the state then has a role in ensuring

201
00:09:51.455 --> 00:09:53.225
that qualifications have currency.

202
00:09:54.215 --> 00:09:58.025
It's up to the state who largely run qualifications

203
00:09:58.125 --> 00:10:02.135
or at least commission agencies

204
00:10:02.135 --> 00:10:03.215
to do that in some cases.

205
00:10:04.925 --> 00:10:07.865
So they need to have currency for them

206
00:10:07.865 --> 00:10:10.465
to be valuable in the labor mar labor market

207
00:10:10.485 --> 00:10:11.825
and in higher education.

208
00:10:12.365 --> 00:10:14.305
So public confidence really matters.

209
00:10:15.085 --> 00:10:17.465
So for example, in England as a regulator

210
00:10:17.845 --> 00:10:20.865
and maintaining public confidence is actually part

211
00:10:20.865 --> 00:10:22.265
of their remit legally.

212
00:10:24.275 --> 00:10:29.175
So this social contract then is much broader than the idea

213
00:10:29.395 --> 00:10:32.895
of using techniques, scientific techniques

214
00:10:33.075 --> 00:10:34.415
to maintain standards

215
00:10:35.075 --> 00:10:38.135
and incorporates wider social practices

216
00:10:38.315 --> 00:10:40.535
and views about what standards mean.

217
00:10:41.595 --> 00:10:45.055
So this then focuses much more on the political

218
00:10:45.195 --> 00:10:46.535
around standards too,

219
00:10:47.155 --> 00:10:49.655
and what's acceptable to a wider society.

220
00:10:50.395 --> 00:10:53.815
And what we saw during the pandemic was a questioning

221
00:10:54.275 --> 00:10:57.375
of the meritocratic basis of the techniques

222
00:10:57.375 --> 00:10:59.615
that were being used and the social,

223
00:11:00.315 --> 00:11:02.375
the social settlement being punctured.

224
00:11:05.335 --> 00:11:09.175
I have to say too, that it's, playing the,

225
00:11:09.285 --> 00:11:12.015
assessments and qualifications can't fix

226
00:11:12.805 --> 00:11:14.535
broader societal issues.

227
00:11:15.475 --> 00:11:19.295
So the tensions around whether they have, ever

228
00:11:20.285 --> 00:11:23.895
been reflective of a meritocratic system have also come

229
00:11:23.915 --> 00:11:27.575
to the fore, far more so during the pandemic.

230
00:11:27.955 --> 00:11:30.215
And I think the, the public discourse around

231
00:11:30.215 --> 00:11:33.815
that is far more, prevalent during the pandemic.

232
00:11:34.125 --> 00:11:37.495
Lots of countries we obviously, had the same issue

233
00:11:37.595 --> 00:11:40.855
to contend with, but countries responded in different ways.

234
00:11:41.835 --> 00:11:46.255
So the most common response in 2020 was actually

235
00:11:46.355 --> 00:11:47.975
to delay the exam.

236
00:11:48.075 --> 00:11:52.375
So for example, in in China, the GCO assessment was delayed

237
00:11:52.475 --> 00:11:53.605
by about a month.

238
00:11:55.165 --> 00:11:56.465
But in the British Isles,

239
00:11:56.685 --> 00:11:58.945
and it's so infrequent they can say that,

240
00:11:58.965 --> 00:12:01.305
but it, I also include Ireland here.

241
00:12:01.525 --> 00:12:05.425
The response was to, use teacher judgements.

242
00:12:05.885 --> 00:12:09.025
And initially the policy was to,

243
00:12:09.465 --> 00:12:12.705
moderate those teacher judgements using the statistical

244
00:12:12.705 --> 00:12:13.745
moderation method.

245
00:12:15.685 --> 00:12:20.305
So, what we saw in, Scotland and England

246
00:12:20.405 --> 00:12:23.365
and, Northern Ireland

247
00:12:23.425 --> 00:12:27.605
and Wales was that there's a, a political change on the day

248
00:12:27.605 --> 00:12:28.805
that results were released

249
00:12:29.225 --> 00:12:32.845
to move away from the statistically moderated grades

250
00:12:32.845 --> 00:12:36.285
that had been issued, back to the teacher,

251
00:12:37.105 --> 00:12:38.285
assessed grades.

252
00:12:39.065 --> 00:12:42.445
And that was because there had been public outcry about

253
00:12:42.795 --> 00:12:44.205
socioeconomic effects

254
00:12:44.825 --> 00:12:47.605
and the sta statistically moderated outcomes

255
00:12:47.605 --> 00:12:49.565
that had been, released.

256
00:12:51.025 --> 00:12:55.605
So, informally, those, moderation procedures came

257
00:12:55.605 --> 00:12:56.885
to be known as the algorithm.

258
00:12:57.865 --> 00:13:00.205
And, the dispute was

259
00:13:00.205 --> 00:13:02.245
around whether there was a postcode lottery

260
00:13:02.945 --> 00:13:04.725
or whether the school that you went to

261
00:13:04.725 --> 00:13:06.645
because of your socioeconomic status

262
00:13:07.225 --> 00:13:09.005
had influenced the results that you

263
00:13:09.005 --> 00:13:10.725
as an individual were given.

264
00:13:11.545 --> 00:13:13.805
So to give a scale of this, the difference

265
00:13:13.805 --> 00:13:16.285
between the statistically moderated results

266
00:13:17.025 --> 00:13:21.405
and the teacher results was, 26% in Scotland

267
00:13:22.065 --> 00:13:24.205
and 36% in England.

268
00:13:24.265 --> 00:13:27.245
So in response to this public outcry, which,

269
00:13:28.055 --> 00:13:29.825
professor Louise Hayward called

270
00:13:30.445 --> 00:13:35.025
the exam results a lightning road for all sorts of,

271
00:13:35.115 --> 00:13:37.785
unfairness in society that came to the fore

272
00:13:38.125 --> 00:13:39.185
during the pandemic.

273
00:13:40.245 --> 00:13:43.385
Um, and result in response to this, po politicians

274
00:13:44.105 --> 00:13:47.065
actually reversed their decisions in 2020

275
00:13:47.365 --> 00:13:49.225
to use the statistical moderation.

276
00:13:49.335 --> 00:13:54.075
However, there's a surprising maintenance

277
00:13:54.175 --> 00:13:57.475
of trust in the exam boards in the face of this.

278
00:13:57.975 --> 00:14:01.355
So in England in 2020, there was still 80%

279
00:14:01.935 --> 00:14:05.075
of stakeholders, trusted

280
00:14:05.945 --> 00:14:07.085
the a-level results.

281
00:14:08.405 --> 00:14:11.985
And in Scotland it was 57% Prior to the pandemic,

282
00:14:12.045 --> 00:14:13.105
it had been 83.

283
00:14:14.445 --> 00:14:18.105
Um, we, we have figures from Scotland that tell us

284
00:14:18.105 --> 00:14:19.145
that three quarters

285
00:14:19.525 --> 00:14:22.585
of stakeholders felt it was the right decision to move

286
00:14:22.585 --> 00:14:23.625
to teacher grades,

287
00:14:24.525 --> 00:14:28.145
but there were lots of question marks about whether,

288
00:14:28.235 --> 00:14:31.505
there would be bias in the teacher grades, given

289
00:14:31.505 --> 00:14:32.825
that teachers know their students,

290
00:14:33.135 --> 00:14:35.225
whether there had been parental pressure

291
00:14:35.225 --> 00:14:36.625
on teachers and so on.

292
00:14:37.125 --> 00:14:39.585
And a bizarre aspect of the pandemic was

293
00:14:39.585 --> 00:14:44.185
that examinations have never been so popular since,

294
00:14:44.415 --> 00:14:45.425
they were removed.

295
00:14:45.645 --> 00:14:49.345
But I think what all of this tells us is that trust

296
00:14:49.375 --> 00:14:50.625
between stakeholders

297
00:14:51.545 --> 00:14:53.805
As necessary to issue exam results.

298
00:14:53.865 --> 00:14:55.765
And that's quite a distributed system.

299
00:14:56.425 --> 00:14:59.805
So that's trust between, trust of the teachers,

300
00:14:59.875 --> 00:15:01.365
it's trust of the exam board,

301
00:15:01.745 --> 00:15:05.445
but it's also trust in schools as institutions not

302
00:15:05.445 --> 00:15:06.485
to compete with each other.

303
00:15:07.185 --> 00:15:08.605
So there were lots of,

304
00:15:08.965 --> 00:15:11.205
question rates about whether the school down the road

305
00:15:11.825 --> 00:15:14.765
had been fair in terms of how they'd graded their students.

306
00:15:15.155 --> 00:15:19.525
This shows us as well that the technocracy can't deal

307
00:15:19.525 --> 00:15:24.165
with trust techno technocracy can put in place systems

308
00:15:24.635 --> 00:15:25.645
that help with that,

309
00:15:26.145 --> 00:15:30.085
but there are much broader social aspects at play here,

310
00:15:30.085 --> 00:15:32.245
which matter for the currency of the exams.

Exploring Exam Standards and Public Perception in Scotland During the Pandemic

00:00:00 - I'm gonna talk specifically to some research we did in Scotland, and the during the pandemic. And the reason for that is that most of the research on what exam standards mean, or high stakes qualification standards mean is actually theoretical. So over the years, we've been developing ideas about the different kinds of conceptualizations people have, about what standards mean. So during the pandemic, this was really my first opportunity

00:00:33 - I had been analyzing the kind of public debates that there have been and so on, and looking at how people conceptualized them. But during the pandemic in Scotland, we were interested in stakeholders views. By that I mean, learners, their parents, teachers, employers, and so on. So we collected data on our theoretical conceptualizations of standards. Now, they weren't gonna understand or use those terms because we have our own technical jargon.

00:01:06 - that everyone could understand. We, we piloted that work so that we knew it was gonna work well. And what we found was that in Scotland, in the sample that we had, so we don't know that that's generalizable to the wider population, but in Scotland, the most popular view of standards was that they represent the performances that students have on the, assessment on the examination. So by that they mean conformed with written criteria

Conceptualizing Standards in Education: Insights from Research

00:01:39 - to do for each grade. Now, the second most popular, way of thinking about standards was that they actually represent, a construct which is underlying the performances, which tells you something more about the individual and is probably more stable. So, for example, you might think of this in terms of ability or intelligence, and about 40% of our sample actually responded in that way.

00:02:10 - but there was also, and these people didn't use standards terms in just one way. So people, agreed with more than one of the items, but about one fifth of the population who responded also said that they thought standards represented the same percentage, getting the grades over time. So we call that, cohort referencing. Some people call it non referencing, which is actually a little bit of a mistake.

00:02:39 - and under normal distribution. So these are the, the kinds of, most popular ways that people talked about standards, though there are others. So one thing to say is that, public understanding of assessment is actually quite low. There are lots of things that are accepted as the way that they're Practiced, when in fact it's, it's actually wrong. Another thing to say is that what we know from the literature is that this isn't just amongst the public.

Global Variations in Assessment Literacy and Practices

00:03:12 - centralized in most, most countries, teachers don't have a lot of assessment literacy either. So this is something that governments around the world are contending with and trying to educate teachers more about. So this means that the public discussions are often at odds with the, the facts about how things are actually done to add to this. understandings of assessment differ across different countries,

00:03:43 - This is something that I noticed when I started going to conferences and hearing people speak about the, controversies in their systems, which just wouldn't have been controversial in my setting or talking about systems that they had that would've been hugely controversial, in the uk. So just to give you a flavor of this, for example, in Malaysia to keep the exams secure, the examiners are held in a compound

00:04:16 - And then between the writing of them and the actual setting of the exams, they're held there. So that's over a month. And this is, it's been shown on tv. The examiners going, in a convoy to this compound. I don't think that would be acceptable practice here in the uk, but there are other examples like how the assessments are done. So use of multiple choice tests with the idea that there's a single right answer for a politics exam in China, that's part of the GCO exam,

Cultural Variations in National Examination Systems

00:04:53 - or in gco until very recently, you needed higher marks in the exam to enter universities. And the most selective universities, if you start the exam in some of the rural constituencies, which again, we wouldn't find, acceptable, but in Ireland, they have exams where the pass marks don't change every year as they do here. And it would be outrageous there if, people wanted

00:05:23 - So to keep that system stable, they actually have to remark some of the exams to make sure that the distributions aren't, very different every year, the distribution of students getting the grades. But if you think about even in the us, the grade point average system that's brought about by teachers marking students on totally different tasks with different criteria all across the country. And those results are used by universities in the US for,

00:05:55 - And just to throw in one more, In Italy, there's a tradition of oral examinations, which again, is so different from what we do here in the uk. So from these examples, what I really want to get across is that national examinations are culturally bound, to the context in which they're conducted. So there are lots of social and historical drivers about what's acceptable

Adapting Assessment Methods During the Pandemic

00:06:24 - and that really affects how people think about what standards are. And research during the pandemic, was much more difficult, obviously, to collect data in the traditional ways that we'd done that. So lots of data collection moved to online, methods. So in Scotland and the research we conducted during the pandemic, we held focus groups, which meant that we were bringing people together from across,

00:06:56 - software like Microsoft Teams. So it's not the case that this was the first time this was done, but it was just being done far more frequently. We also, conducted a survey, with nearly so altogether, there were over a thousand people involved in that research. All of it conducted online. And really these methods that we used during the pandemic and are more reliance on technology to do these, these forms

00:07:26 - When we talk about national standards within the assessment community, we have to, we've had to over the years, construct rules so that we can make things fairer. We have lots of ways in which, the standards are checked so that ensure the marking is fair and so on that they question papers reflect the curriculum and all the rest of it. Now, to do that, it means that we've had

Challenges of Jargon and Public Understanding in Assessment Systems

00:07:59 - So we have jargon in relation to this. In other words, we've had to codify the field. Now that assessment expertise then is quite concentrated in assessment organizations. And it means that when we talk to people outside that field, they may be using different language and they may not understand, how these terms reflect quite complex processes, underline the national assessments.

00:08:30 - that we are a techno, a technocracy, that we've set up this edifice of expertise with our own language and that we're insiders speaking to ourselves. And that quite often some of the terms we, we use don't address public concerns, in the terms that are, they're being used, publicly. The thing about the pandemic was this punctured the social settlement that we had around standards

00:09:02 - that were being raised, in the terms that they were being raised. Now it is, well, for a long time it's been known that there are always tension and contested issues around examinations because they affect people's lives chances to a large extent. and that social settlement that we normally have, right, relies on a system of meritocracy in which people give up their individual rights

Social Contracts and Public Confidence in Qualifications

00:09:34 - So that's the kind of social contracts that I'm, I'm talking about that there are wider societal benefits of having a system, that is, intentionally metastatic. So the state then has a role in ensuring that qualifications have currency. It's up to the state who largely run qualifications or at least commission agencies

00:10:04 - So they need to have currency for them to be valuable in the labor mar labor market and in higher education. So public confidence really matters. So for example, in England as a regulator and maintaining public confidence is actually part of their remit legally. So this social contract then is much broader than the idea of using techniques, scientific techniques

00:10:35 - and incorporates wider social practices and views about what standards mean. So this then focuses much more on the political around standards too, and what's acceptable to a wider society. And what we saw during the pandemic was a questioning of the meritocratic basis of the techniques that were being used and the social, the social settlement being punctured.

Pandemic Disruptions to Assessment and Meritocracy

00:11:09 - assessments and qualifications can't fix broader societal issues. So the tensions around whether they have, ever been reflective of a meritocratic system have also come to the fore, far more so during the pandemic. And I think the, the public discourse around that is far more, prevalent during the pandemic. Lots of countries we obviously, had the same issue

00:11:41 - So the most common response in 2020 was actually to delay the exam. So for example, in in China, the GCO assessment was delayed by about a month. But in the British Isles, and it's so infrequent they can say that, but it, I also include Ireland here. The response was to, use teacher judgements. And initially the policy was to,

00:12:12 - moderation method. So, what we saw in, Scotland and England and, Northern Ireland and Wales was that there's a, a political change on the day that results were released to move away from the statistically moderated grades that had been issued, back to the teacher, assessed grades. And that was because there had been public outcry about

Assessment Fairness and Public Trust During the Pandemic

00:12:44 - and the sta statistically moderated outcomes that had been, released. So, informally, those, moderation procedures came to be known as the algorithm. And, the dispute was around whether there was a postcode lottery or whether the school that you went to because of your socioeconomic status had influenced the results that you as an individual were given. So to give a scale of this, the difference

00:13:17 - and the teacher results was, 26% in Scotland and 36% in England. So in response to this public outcry, which, professor Louise Hayward called the exam results a lightning road for all sorts of, unfairness in society that came to the fore during the pandemic. and result in response to this, po politicians

00:13:47 - to use the statistical moderation. However, there's a surprising maintenance of trust in the exam boards in the face of this. So in England in 2020, there was still 80% of stakeholders, trusted the a-level results. And in Scotland it was 57% Prior to the pandemic, it had been 83.

Trust and Controversy in Teacher-Assigned Grades During the Pandemic

00:14:18 - that three quarters of stakeholders felt it was the right decision to move to teacher grades, but there were lots of question marks about whether, there would be bias in the teacher grades, given that teachers know their students, whether there had been parental pressure on teachers and so on. And a bizarre aspect of the pandemic was that examinations have never been so popular since, they were removed.

00:14:49 - between stakeholders As necessary to issue exam results. And that's quite a distributed system. So that's trust between, trust of the teachers, it's trust of the exam board, but it's also trust in schools as institutions not to compete with each other. So there were lots of, question rates about whether the school down the road had been fair in terms of how they'd graded their students. This shows us as well that the technocracy can't deal

00:15:30 - that help with that, but there are much broader social aspects at play here, which matter for the currency of the exams.

Faculti AI can make mistakes

Research

Policy

Analysis

Data

Originals

Explainers

Podcasts

Faculti

New on Faculti

Idea Explorer

Definition

News Context

Critical Questions and Answers

Research Summaries

Standards as a social contract in curriculum-based qualifications

Media Literacy Toolkit

Critical Questions to Consider ▼

Bias and Perspective Awareness ▼

Further Reading ▼