Popular in Professional

Business

The Allegory of the Favela: The Multifaceted Effects of Socioeconomic Mobility

University of Pennsylvania Leandro S. Pongeluppe

Architecture

Connecting Cities and Nature

University of Virginia Timothy Beatley

Human Rights Law

Human Rights and Corruption: Problems and Potential of Individualising a Systemic Problem

Max Planck Institute Anne Peters

Finance

Chinese Exchange Rate Policy: Lessons for Global Investors

University of California San Diego Michael Melvin

Education

Firm Performance Pay as Insurance against Promotion Risk

Stockholm School of Economics Alvin Chen

Constitutional Law

What is constitutional ideology?

University of Oxford Nick Barber

A statistical approach to law school citation rankings

View more insights in Professional - Law

View more figures from University of Virginia

View the page of Joshua Fischman

Read the Study

Next in Law

The Discursive Evolution of Human Rights Law

Renana Keydar, Vera Shikhelman | Hebrew University of Jerusalem

Affirmatively Furthering Health Equity

Mary Crossley | University of Pittsburgh

Do citation rankings truly reflect the scholarly impact of law schools, or are they a statistical mirage with little informational value on faculty quality? Joshua Fischman discusses the statistical underpinnings and substantial uncertainties inherent in law school citation rankings through a novel approach modeling citations as lognormally distributed, questioning the efficacy of common ranking methods.

Image courtesy of interviewee. November 6, 2024

Log-in or Sign-up to Faculti
Currently viewing this subject insight as a guest. You have insight(s) remaining for this month. Login to view 8000+ figures on the platform.

Generating Workspace...

Export as PDF

This video discusses the complexities and challenges surrounding law school citation rankings. It highlights that many law schools are closely clustered in terms of citation counts, making small differences often seem arbitrary and insignificant. The author criticizes previous approaches for lacking meaningful criteria and advocates for empirically assessing whether differences in citations reflect genuine disparities in scholarly quality.

The distribution of citations is predominantly log-normal, leading to skewed results where a few "superstar" professors overshadow their peers. The author proposes using the geometric mean for more accurate comparisons, arguing it is less sensitive to outliers and better represents the overall faculty quality.

The discussion also touches on how citation counts can potentially misguide prospective students, as they may place undue importance on minor rank differences. Furthermore, the author warns that law schools may try to manipulate citations by hiring faculty in more highly cited fields, which can compromise academic integrity.

Finally, while acknowledging that citation metrics have some predictive value regarding faculty career advancements, the author suggests combining them with other measures, such as publication venues and quality indicators, to better reflect scholarly merit. This video concludes by urging consideration of when differences in citations are significant, emphasizing the need for statistical rigor in understanding and utilizing these rankings.

Language:

Export as PDF

Export as PDF
Here are five key points derived from the video regarding law school citation rankings and their implications:

1. **Arbitrariness of Citation Rankings**: The process of ranking law schools based on citations is often arbitrary, lacking clear criteria to determine meaningful differences. Many schools are closely clustered in citation counts, making small differences statistically insignificant.

2. **Nature of Citations**: Citations are seen as a noisy measure of scholarly impact and do not directly correlate with the quality of scholarship. The video argues that while citations reflect some level of recognition, they should not be treated as the primary objective of scholarship.

3. **Potential for Ranking Manipulation**: Law schools may manipulate citation rankings by hiring faculty in more cited fields or seeking well-cited senior professors, which can distort the perceived scholarly strength of a faculty and negatively affect academic integrity.

4. **Distribution of Citations**: Citation counts follow a log-normal distribution, indicating that a few 'superstar' professors dominate citation counts. This skew necessitates careful statistical analysis to compare faculties beyond simple averages.

5. **Need for Improved Metrics**: The author suggests that relying solely on citation counts can be misleading. It is essential to incorporate additional metrics and consider the broader contributions of faculty members, as well as the convideo of their work, to develop more informative measures for evaluating scholarly quality.

Export as PDF
Here are some key quotes extracted from the video regarding law school citation rankings:

1. **Clustered Rankings**:
- "Many law schools are very closely clustered together... small differences don't really matter... it was really all just completely arbitrary approaches to determining whether differences mattered or not."

2. **Purpose of Scholarship**:
- “Scholarship is not a game where whoever gets the most citations wins... You’re trying to actually advance knowledge.”

3. **Data Generating Process**:
- “Can we look at those signals and think about the data generating process... and ask, well, are these differences really significant?”

4. **Statistical Distribution**:
- “I found that the citations were logged, normally distributed... the log normal distribution is extremely skewed.”

5. **Exponential Growth of Citations**:
- “Citations beget citations, and citation counts grow exponentially.”

6. **Equality and Citations**:
- "Ultimately we need to think about how this relates to what we think of as equality... citations are in some sense, exponential for scholar equality."

7. **Meaningful Differences**:
- “The problem is, if the differences among faculties are mostly statistical noise... it can lead to misleading results.”

8. **Gaming Rankings**:
- “Law schools put a lot of effort into gaming the rankings... deans get fired because they drop one or two places in the rankings.”

9. **Challenges in Measuring Impact**:
- “Not everyone agrees with looking at career advancement... not everyone agrees with the choices that Harvard, Yale and Stanford make in terms of who they hire.”

10. **Methodological Critique**:
- “There's lots of ways to count citations that will give us different results... we have to think carefully.”

11. **Geometric Mean Advantage**:
- “The geometric mean... better captures the whole range of members on each faculty... much less sensitive to the outliers.”

12. **Indistinguishable Schools**:
- “The schools in the middle, you basically can't tell them apart... for most of the schools in the middle, there wouldn't be a huge difference.”

13. **Database Limitations**:
- “Every database has its advantages and its problems... I don’t think we can really rely on theory to tell us which database to use.”

14. **Disciplinary Differences in Citation Practices**:
- “Each discipline has its own practices... not all of which are meaningful.”

15. **Noise of Citations**:
- “Everyone should remember that citations are noisy signals of scholarly impact... we should be thinking about statistical methods to determine whether differences are meaningful or not.”

These quotes encapsulate the main arguments and insights regarding the complexities and issues surrounding law school citation rankings.

X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:90000

1
00:00:00.205 --> 00:00:03.005
A general issue with law school citation rankings

2
00:00:03.385 --> 00:00:04.765
is law schools.

3
00:00:04.915 --> 00:00:07.285
Many law schools are very closely clustered together.

4
00:00:07.705 --> 00:00:09.685
And anyone in the past who had ever published

5
00:00:10.225 --> 00:00:12.885
law school citation citation rankings is acknowledged.

6
00:00:13.185 --> 00:00:15.005
Oh, small differences don't really matter.

7
00:00:15.465 --> 00:00:16.685
Um, but there were no criteria

8
00:00:17.025 --> 00:00:19.085
for determining whether differences are meaningful or not.

9
00:00:19.785 --> 00:00:22.965
It was really all just completely arbitrary approaches

10
00:00:22.985 --> 00:00:25.845
to determining whether differences mattered or not.

11
00:00:26.035 --> 00:00:29.925
Whereas in most of legal scholarship, we actually look

12
00:00:29.945 --> 00:00:32.405
to see do, or empirical scholarship,

13
00:00:32.445 --> 00:00:33.685
I should say empirical legal scholarship.

14
00:00:34.425 --> 00:00:36.925
Are differences significant or estimates significant?

15
00:00:37.665 --> 00:00:39.365
Um, can you actually look from the data

16
00:00:39.385 --> 00:00:40.925
and say that this tells us

17
00:00:41.315 --> 00:00:43.485
that there's some underlying measure

18
00:00:44.115 --> 00:00:47.005
that is actually distinct between say, Harvard Law School

19
00:00:47.385 --> 00:00:48.565
and some other law school?

20
00:00:48.785 --> 00:00:51.845
We should think about the citations not

21
00:00:51.945 --> 00:00:53.885
as the object of scholarship, right?

22
00:00:53.885 --> 00:00:55.005
Scholarship is not a game

23
00:00:55.375 --> 00:00:56.965
where whoever gets the most citations

24
00:00:56.965 --> 00:00:58.925
wins, you know, scholarship.

25
00:00:58.925 --> 00:01:00.205
You're trying to actually advance knowledge.

26
00:01:00.205 --> 00:01:04.865
And citations are a noisy measure of, you know,

27
00:01:04.865 --> 00:01:07.065
whether you're having impact, whether you have good ideas.

28
00:01:07.285 --> 00:01:08.945
And so I was thinking more about

29
00:01:08.945 --> 00:01:09.985
the data generating process.

30
00:01:10.085 --> 00:01:13.465
You know, how do you know faculties, hire professors,

31
00:01:13.465 --> 00:01:15.945
professors write articles.

32
00:01:15.945 --> 00:01:17.585
Those articles get cited.

33
00:01:18.125 --> 00:01:20.825
Um, and that gives us a signal about the quality,

34
00:01:20.965 --> 00:01:22.425
the scholarly quality of the professor,

35
00:01:22.725 --> 00:01:25.705
and the qual the scholarly quality of the faculty.

36
00:01:26.285 --> 00:01:28.385
And then can we look at those signals

37
00:01:28.885 --> 00:01:30.505
and think about the data generating process

38
00:01:30.565 --> 00:01:31.945
and the statistical distribution

39
00:01:32.605 --> 00:01:36.025
and ask, well, are these differences really significant?

40
00:01:36.045 --> 00:01:39.305
Do they tell us that the average quality

41
00:01:39.365 --> 00:01:42.145
of a professor on one faculty is actually higher than the

42
00:01:42.145 --> 00:01:43.745
average quality another faculty?

43
00:01:43.885 --> 00:01:45.465
And no one had really tried to do that before.

44
00:01:46.355 --> 00:01:49.735
Um, so I found that the citations were logged,

45
00:01:49.935 --> 00:01:51.455
normally distributed,

46
00:01:51.635 --> 00:01:55.495
and that was an important part of, of this project.

47
00:01:56.395 --> 00:01:57.495
Um, and in part

48
00:01:57.495 --> 00:02:00.255
because the log normal distribution is extremely skewed.

49
00:02:00.955 --> 00:02:03.535
Um, so superstar professors look like they're

50
00:02:03.535 --> 00:02:05.495
so much more important than everyone else.

51
00:02:06.115 --> 00:02:08.055
Um, but the fact is, and there is a literature on this from

52
00:02:08.055 --> 00:02:11.855
other fields that, citations kind of accumulate

53
00:02:11.855 --> 00:02:13.855
through an exponential process, right?

54
00:02:13.875 --> 00:02:16.055
You, you publish an article, it gets cited.

55
00:02:16.055 --> 00:02:17.055
And when it gets cited, that means

56
00:02:17.055 --> 00:02:18.255
more people discover your work.

57
00:02:18.595 --> 00:02:19.775
And the more people discover your work,

58
00:02:19.775 --> 00:02:20.815
the more they cite it.

59
00:02:21.355 --> 00:02:23.815
Um, and then it, I said, it grows exponentially,

60
00:02:23.815 --> 00:02:25.335
and then it also enhances your reputation.

61
00:02:25.715 --> 00:02:27.735
So when you publish new articles,

62
00:02:27.915 --> 00:02:29.495
people immediately wanna read them

63
00:02:29.495 --> 00:02:30.695
because they know your reputation.

64
00:02:30.795 --> 00:02:33.055
And so citations beget citations,

65
00:02:33.395 --> 00:02:35.695
and citation counts grow exponentially.

66
00:02:36.435 --> 00:02:38.655
Um, that exponential rate may be faster

67
00:02:38.715 --> 00:02:40.695
for some scholars than for others based on

68
00:02:40.995 --> 00:02:42.295
the contributions in their work.

69
00:02:42.955 --> 00:02:47.035
Um, but ultimately we need to think about how this relates

70
00:02:47.055 --> 00:02:49.235
to what we think of as equality.

71
00:02:49.615 --> 00:02:51.395
And I do think that citations are in some sense,

72
00:02:51.395 --> 00:02:53.515
exponential for scholar equality.

73
00:02:53.615 --> 00:02:56.395
And then the log normal distribution helps us both

74
00:02:56.395 --> 00:02:58.355
because it gives us a statistical test

75
00:02:58.815 --> 00:03:00.445
for comparing the two faculties.

76
00:03:00.445 --> 00:03:03.205
We can ask, does it look like their citations are drawn from

77
00:03:03.205 --> 00:03:06.125
the same distribution, or does one faculty's distribution

78
00:03:06.395 --> 00:03:07.725
look like it dominates another?

79
00:03:08.465 --> 00:03:11.325
Um, and it also gives us the measure, the geometric mean

80
00:03:11.865 --> 00:03:16.165
for essentially averaging the citations within each faculty.

81
00:03:16.905 --> 00:03:18.245
Um, and I think that's important

82
00:03:18.245 --> 00:03:21.885
because, people have been using very arbitrary measures.

83
00:03:21.885 --> 00:03:23.205
Haven't thought about why, given

84
00:03:23.205 --> 00:03:26.285
that the differences in the citation counts are mostly

85
00:03:26.355 --> 00:03:30.685
insignificant for any kind of faculties that are thought of

86
00:03:30.685 --> 00:03:32.365
as peers or comparable.

87
00:03:33.275 --> 00:03:35.605
I'll be honest, I don't think prospective students

88
00:03:35.625 --> 00:03:38.005
or should be giving any weight at all

89
00:03:38.465 --> 00:03:40.365
to small differences in citation rankings.

90
00:03:40.745 --> 00:03:44.975
Now, now, I'm gonna be honest as well. Um, faculty may care.

91
00:03:45.135 --> 00:03:47.215
I don't think most prospective students really care

92
00:03:47.215 --> 00:03:50.055
that much about how many citations their perspective,

93
00:03:50.635 --> 00:03:51.735
professors have.

94
00:03:52.595 --> 00:03:55.695
Um, but they do give an enormous amount of weight

95
00:03:55.835 --> 00:03:57.975
to the rankings that are published by US News.

96
00:03:58.555 --> 00:04:01.415
Um, now those are focused more on employment outcomes.

97
00:04:01.515 --> 00:04:03.015
Do graduates pass the bar

98
00:04:03.015 --> 00:04:05.135
and get good jobs to some degree

99
00:04:05.135 --> 00:04:06.775
incoming students PR credentials?

100
00:04:06.845 --> 00:04:09.455
Will my peers, have good credentials?

101
00:04:10.135 --> 00:04:11.855
I think students also care about some things

102
00:04:11.855 --> 00:04:12.855
outside the rankings.

103
00:04:12.855 --> 00:04:15.855
Are the students happy? Do professors care about teaching?

104
00:04:15.855 --> 00:04:17.335
That sort of thing. Um,

105
00:04:17.835 --> 00:04:20.295
so I think citation rankings have been something

106
00:04:20.295 --> 00:04:23.215
that faculty are more fixated on than students right now.

107
00:04:23.835 --> 00:04:26.295
Um, but they do matter for very important reason.

108
00:04:26.355 --> 00:04:28.015
And that is that there's speculation

109
00:04:28.015 --> 00:04:31.055
that they could be incorporated into general rankings like

110
00:04:31.055 --> 00:04:32.135
the US News rankings.

111
00:04:32.715 --> 00:04:34.775
Um, US news even announced several years ago

112
00:04:34.775 --> 00:04:36.415
that they were gonna publish a

113
00:04:36.615 --> 00:04:37.775
citation ranking of law schools.

114
00:04:38.285 --> 00:04:40.655
They abandoned that plan, at least the time being in part

115
00:04:40.655 --> 00:04:42.175
because they couldn't figure out how to do it, right.

116
00:04:42.645 --> 00:04:43.855
It's not that easy thing to do,

117
00:04:43.875 --> 00:04:46.415
but there's certainly a potential it could be revived.

118
00:04:46.715 --> 00:04:49.215
Um, US News has historically,

119
00:04:49.575 --> 00:04:50.655
measured the scholarly reputation

120
00:04:50.655 --> 00:04:52.415
of faculties using surveys,

121
00:04:52.915 --> 00:04:54.215
but that's been widely criticized.

122
00:04:54.215 --> 00:04:55.855
The average professor really isn't following

123
00:04:55.855 --> 00:04:57.735
that much what's going on at other schools.

124
00:04:58.555 --> 00:05:00.295
Um, and so it seems to just sort

125
00:05:00.295 --> 00:05:01.495
of perpetuate the hierarchy.

126
00:05:01.495 --> 00:05:03.175
And so a lot of people have proposed citation

127
00:05:03.175 --> 00:05:04.975
rankings as an alternative.

128
00:05:05.755 --> 00:05:08.405
Um, but the problem is, if the differences

129
00:05:08.405 --> 00:05:10.565
among faculties are mostly statistical noise,

130
00:05:10.955 --> 00:05:12.965
this generates some problems.

131
00:05:12.985 --> 00:05:15.165
You know, first it can lead to misleading results.

132
00:05:15.665 --> 00:05:17.485
The rankings could amplify

133
00:05:17.995 --> 00:05:19.525
what are really meaningless differences.

134
00:05:20.345 --> 00:05:23.525
Um, but the second problem is that law schools put a lot

135
00:05:23.525 --> 00:05:25.245
of effort into gaming the rankings.

136
00:05:25.945 --> 00:05:28.285
Um, students care about the rankings

137
00:05:28.285 --> 00:05:30.205
and law schools need to make sure that they're

138
00:05:30.465 --> 00:05:31.605
as highly ranked as they can.

139
00:05:31.735 --> 00:05:34.405
Deans get fired because they, they drop one

140
00:05:34.405 --> 00:05:35.845
or two places in the rankings.

141
00:05:36.505 --> 00:05:39.565
Um, and so they could also try to game citation rankings.

142
00:05:40.105 --> 00:05:41.765
Um, you can any law school can,

143
00:05:41.765 --> 00:05:43.005
IM improve its citation count

144
00:05:43.505 --> 00:05:46.765
by hiring faculty in widely cited fields like constitutional

145
00:05:46.825 --> 00:05:50.365
law, avoid tax or specialty fields,

146
00:05:50.385 --> 00:05:53.285
or interdisciplinary work fields are cited less.

147
00:05:53.985 --> 00:05:56.685
Um, they could hire more senior professors who have lots

148
00:05:56.685 --> 00:05:58.805
of citations and avoid hiring young people.

149
00:05:59.465 --> 00:06:01.405
Um, now that could be good for school's rankings,

150
00:06:01.405 --> 00:06:02.605
but that's not good for the academy.

151
00:06:02.795 --> 00:06:04.965
It's not good for scholarship. It's not good for teaching.

152
00:06:05.505 --> 00:06:07.565
Um, and so I wanna be clear if,

153
00:06:07.625 --> 00:06:09.885
if we're really talking about incorporating this into a

154
00:06:09.885 --> 00:06:11.645
broader ranking, this could lead to a lot of gaming

155
00:06:12.305 --> 00:06:13.765
and it would be especially problematic

156
00:06:13.845 --> 00:06:15.565
'cause it doesn't seem like it's very informative.

157
00:06:15.765 --> 00:06:18.605
I have a another paper with my colleague Michael Livermore,

158
00:06:18.905 --> 00:06:22.085
where we're looking at how well citation ranking,

159
00:06:22.745 --> 00:06:25.565
or I'm sorry, citation counts, predict career advancements.

160
00:06:25.565 --> 00:06:28.245
So for example, do professors move,

161
00:06:28.785 --> 00:06:30.965
get jobs at more prestigious law schools?

162
00:06:31.815 --> 00:06:33.285
Professors who are highly cited.

163
00:06:33.745 --> 00:06:36.205
And we find that it is has some predictive value,

164
00:06:36.385 --> 00:06:39.245
but you get a lot more predictive value if you look at other

165
00:06:39.245 --> 00:06:40.445
measures besides citation.

166
00:06:40.585 --> 00:06:42.205
So for example,

167
00:06:42.865 --> 00:06:44.685
do people publish in peer review journals?

168
00:06:44.685 --> 00:06:46.365
Do they publish in prestigious journals?

169
00:06:47.145 --> 00:06:50.205
Um, do they, author books that are published

170
00:06:50.205 --> 00:06:51.765
by prestigious university presses?

171
00:06:52.345 --> 00:06:54.845
So if you incorporate something that was a little broader,

172
00:06:55.785 --> 00:06:57.245
we find

173
00:06:57.245 --> 00:06:59.205
that actually predicts career advancement better.

174
00:06:59.425 --> 00:07:02.365
Now, not everyone agrees with looking at career advancement.

175
00:07:02.425 --> 00:07:04.285
You know, not everyone agrees with the choices that Harvard,

176
00:07:04.355 --> 00:07:06.245
Yale and Stanford make in terms of who they hire.

177
00:07:06.945 --> 00:07:09.725
Um, but you do find that those top schools hire a lot

178
00:07:09.725 --> 00:07:11.485
of people who don't have high citation counts.

179
00:07:11.715 --> 00:07:13.325
They might be legal historians,

180
00:07:13.395 --> 00:07:15.685
they might be people in niche areas

181
00:07:15.865 --> 00:07:17.285
and new fields that are emerging

182
00:07:17.335 --> 00:07:19.045
where there aren't a lot of citations yet.

183
00:07:19.705 --> 00:07:22.125
Um, and so there could be a, a bit of a bias.

184
00:07:22.665 --> 00:07:25.565
Um, and in legal scholarship in particular, you know,

185
00:07:25.975 --> 00:07:28.765
large articles could have 500 footnotes

186
00:07:28.765 --> 00:07:30.205
with thousands of citations.

187
00:07:30.625 --> 00:07:32.285
And not every one of them is really meaningful.

188
00:07:32.285 --> 00:07:35.165
It's not like physics where you have a few dozen citations

189
00:07:35.465 --> 00:07:37.165
and you're really giving intellectual credit,

190
00:07:37.905 --> 00:07:39.165
to the other authors.

191
00:07:39.165 --> 00:07:40.565
Some of them are not that meaningful.

192
00:07:41.265 --> 00:07:44.165
And sometimes people get

193
00:07:44.165 --> 00:07:45.485
cited 'cause they have brilliant new ideas.

194
00:07:45.545 --> 00:07:46.925
And sometimes people get cited

195
00:07:46.925 --> 00:07:49.925
because they wrote a well-written survey article.

196
00:07:50.705 --> 00:07:52.445
Um, that is the go-to article.

197
00:07:52.545 --> 00:07:54.445
If you're writing about something in particular,

198
00:07:54.725 --> 00:07:56.685
I would be a little concerned if we just blindly counted

199
00:07:57.005 --> 00:07:59.805
citations without thinking about how they correspond

200
00:07:59.805 --> 00:08:01.845
to the way we perceive merit with, with the caveat

201
00:08:01.845 --> 00:08:04.205
that we don't agree on what merit is.

202
00:08:04.205 --> 00:08:06.205
Some people might really like survey articles more than they

203
00:08:06.205 --> 00:08:07.485
like interdisciplinary work.

204
00:08:07.505 --> 00:08:08.965
You know, I, I don't,

205
00:08:08.965 --> 00:08:10.965
but that's, you know, there are some,

206
00:08:10.995 --> 00:08:12.005
some disagreements there.

207
00:08:12.105 --> 00:08:13.685
But I do think we have to be careful

208
00:08:13.865 --> 00:08:15.845
and there's lots of ways to count citations

209
00:08:16.105 --> 00:08:18.645
and different citation counting methods

210
00:08:18.675 --> 00:08:19.845
will give us different results.

211
00:08:20.145 --> 00:08:21.685
And we have to think carefully.

212
00:08:21.685 --> 00:08:23.045
If we incorporate anything into the

213
00:08:23.045 --> 00:08:24.125
rankings, how would we do it?

214
00:08:24.705 --> 00:08:27.685
If you are comparing the citations on two different

215
00:08:27.685 --> 00:08:29.685
faculties, you need some measure

216
00:08:29.785 --> 00:08:31.685
to summarize the citations on each faculty.

217
00:08:32.425 --> 00:08:36.525
Now, a pioneering study of law school citation counts

218
00:08:36.525 --> 00:08:39.365
that actually had a sort of rudimentary ranking from 25

219
00:08:39.365 --> 00:08:41.565
years ago, thought about this question.

220
00:08:41.635 --> 00:08:43.205
They said, you know, you could use the mean,

221
00:08:43.265 --> 00:08:44.605
you could use the median.

222
00:08:45.345 --> 00:08:47.685
Um, they couldn't decide which is better.

223
00:08:47.685 --> 00:08:49.485
And so they ended up just taking the sum of the mean

224
00:08:49.485 --> 00:08:52.125
and the median as a kind of throwing their hands up in the

225
00:08:52.125 --> 00:08:53.645
air, what measures should I use?

226
00:08:53.705 --> 00:08:58.645
And then, later on, another ranking by, a professor

227
00:08:58.645 --> 00:09:01.725
who has a very, prominent online presence decided

228
00:09:01.725 --> 00:09:03.685
to use twice the mean plus the median.

229
00:09:04.145 --> 00:09:07.285
Um, 'cause I think, well, I don't really know why of

230
00:09:08.065 --> 00:09:09.485
it was still pretty arbitrary.

231
00:09:09.505 --> 00:09:11.005
And then others started to follow that.

232
00:09:11.225 --> 00:09:13.485
And, you know, there's studies that actually would say,

233
00:09:13.955 --> 00:09:15.605
this measure doesn't really make any sense,

234
00:09:15.665 --> 00:09:16.845
but everyone else is using it.

235
00:09:16.845 --> 00:09:18.085
And we wanna be consistent with the

236
00:09:18.245 --> 00:09:19.365
literature, so we're using it.

237
00:09:19.905 --> 00:09:23.805
Um, and that's a terrible way to rank law schools.

238
00:09:23.825 --> 00:09:25.245
And of course, the rankings look different

239
00:09:25.245 --> 00:09:26.485
depending on which measure you use.

240
00:09:27.145 --> 00:09:28.445
Um, and if you can't give a reason

241
00:09:28.445 --> 00:09:29.965
for choosing one me measure over another,

242
00:09:30.355 --> 00:09:31.805
then you really can't give a reason

243
00:09:31.825 --> 00:09:33.965
for choosing one ranking over another.

244
00:09:34.585 --> 00:09:35.805
Um, that necessarily means

245
00:09:35.805 --> 00:09:37.045
there'll be a lot of arbitrariness.

246
00:09:37.425 --> 00:09:42.305
So, my choice of geometric mean was motivated

247
00:09:42.305 --> 00:09:44.345
by the distribution of the citations.

248
00:09:44.365 --> 00:09:47.585
The fact that you have, log normal distribution.

249
00:09:48.285 --> 00:09:52.445
Um, and the geometric mean is essentially the most efficient

250
00:09:52.445 --> 00:09:55.565
measure of central tendency for the log normal distribution.

251
00:09:55.665 --> 00:09:57.205
So it's essentially sort

252
00:09:57.205 --> 00:10:00.205
of squeezing the most information out of the data,

253
00:10:00.695 --> 00:10:02.765
gives us the best estimate of central tendency.

254
00:10:02.835 --> 00:10:04.925
Also gives us very easy statistical tests

255
00:10:05.105 --> 00:10:08.005
to see if the citations from two faculties were drawn from

256
00:10:08.005 --> 00:10:09.405
the same distribution or not.

257
00:10:10.145 --> 00:10:13.885
Um, now another practical effect of the geometric mean is

258
00:10:13.885 --> 00:10:15.845
that it's much less sensitive to the outliers.

259
00:10:16.695 --> 00:10:18.875
Um, as I mentioned, it's a very skewed distribution.

260
00:10:19.215 --> 00:10:21.195
And if you just take the regular arithmetic mean,

261
00:10:21.855 --> 00:10:24.435
it really is enormously influenced

262
00:10:24.495 --> 00:10:27.355
by the presses at the very top of the distribution.

263
00:10:27.815 --> 00:10:29.635
As I mentioned, you know, the citations accumulate

264
00:10:29.635 --> 00:10:30.635
through an exponential process.

265
00:10:31.215 --> 00:10:32.475
And so that means that the very top

266
00:10:32.475 --> 00:10:34.195
professors really dominate.

267
00:10:34.575 --> 00:10:35.755
And so, lemme give you an example.

268
00:10:36.135 --> 00:10:39.235
Um, the most cited professor in my data was Professor

269
00:10:39.255 --> 00:10:40.755
Casson Stein at Harvard Law School.

270
00:10:41.215 --> 00:10:42.275
Um, in the last five years,

271
00:10:42.295 --> 00:10:44.595
he had 5,800 citations a little bit more.

272
00:10:45.335 --> 00:10:48.355
Um, and that's about 24 times as many citations

273
00:10:48.695 --> 00:10:50.275
as the median faculty member at Harvard.

274
00:10:51.095 --> 00:10:52.795
Um, now Cass is a superstar.

275
00:10:52.875 --> 00:10:54.075
I have enormous respect for him.

276
00:10:54.575 --> 00:10:57.035
Um, but I would be a little skeptical of any measure

277
00:10:57.035 --> 00:10:58.955
that values him 24 times as much

278
00:10:59.135 --> 00:11:02.475
as a typical Harvard Law School, faculty member.

279
00:11:03.215 --> 00:11:05.035
Um, and so if you're using a measure

280
00:11:05.035 --> 00:11:06.795
that's based on the arithmetic meme,

281
00:11:06.855 --> 00:11:07.915
you're really looking at

282
00:11:08.055 --> 00:11:10.275
how many cast sunscreens does each faculty have.

283
00:11:11.015 --> 00:11:13.555
Um, and maybe there's some people who care about that.

284
00:11:13.555 --> 00:11:14.995
They only wanna know how many superstars

285
00:11:14.995 --> 00:11:16.115
are there on a particular faculty.

286
00:11:16.875 --> 00:11:18.355
Although, once again, you have to both a superstar

287
00:11:18.355 --> 00:11:20.195
and in the highly sighted field,

288
00:11:20.195 --> 00:11:22.515
attack Superstar is still gonna look pretty adorable.

289
00:11:23.295 --> 00:11:26.475
Um, but I think most stake care stakeholders really care

290
00:11:26.475 --> 00:11:29.275
about the entire faculty, not just the most famous.

291
00:11:29.775 --> 00:11:31.275
If you're a student, you're not gonna take all your

292
00:11:31.275 --> 00:11:32.435
classes with Cass Sunstein.

293
00:11:33.015 --> 00:11:36.515
Uh, and so the geometric mean,

294
00:11:36.715 --> 00:11:39.715
I think better captures the whole range

295
00:11:40.055 --> 00:11:41.115
of members on each faculty.

296
00:11:41.215 --> 00:11:43.595
So that's sort of another intuitive advantage.

297
00:11:43.875 --> 00:11:45.315
I think the wide confidence, and it'll show

298
00:11:45.315 --> 00:11:47.115
that these rankings don't really convey

299
00:11:47.115 --> 00:11:48.195
a lot of useful information.

300
00:11:49.095 --> 00:11:53.155
Um, but it is possible that we could devise other measures

301
00:11:53.245 --> 00:11:54.485
that are more useful.

302
00:11:54.825 --> 00:11:57.605
Um, you know, I mentioned earlier, when we were trying

303
00:11:57.605 --> 00:11:59.485
to predict whether, you know,

304
00:11:59.505 --> 00:12:01.925
people get jobs at better law schools, looking at

305
00:12:02.175 --> 00:12:05.205
where they publish, looking at books,

306
00:12:05.255 --> 00:12:08.245
there might be other kinds of scholarly indicia

307
00:12:08.245 --> 00:12:09.325
that would be relevant.

308
00:12:09.865 --> 00:12:12.685
Um, maybe awards, you know, you'd have to think about it,

309
00:12:12.745 --> 00:12:15.165
but it's possible if you combine citations

310
00:12:15.595 --> 00:12:16.605
with other measures,

311
00:12:16.825 --> 00:12:19.325
you might get something a little bit more informative.

312
00:12:19.325 --> 00:12:21.965
And the same is true, I think, if you account for seniority.

313
00:12:22.705 --> 00:12:26.965
Uh, but I'm a little skeptical

314
00:12:26.965 --> 00:12:28.805
that we're gonna get anything that's really precise.

315
00:12:28.805 --> 00:12:31.045
These contents intervals were so wide.

316
00:12:31.245 --> 00:12:33.565
I mean, shockingly wide. I had no idea when I started this

317
00:12:33.565 --> 00:12:34.485
project, I was

318
00:12:34.485 --> 00:12:35.565
gonna get, you know, the schools in the middle.

319
00:12:35.565 --> 00:12:37.045
You basically can't tell them apart.

320
00:12:37.045 --> 00:12:39.685
There's a a hundred some schools that are more

321
00:12:39.685 --> 00:12:43.325
or less indistinguishable, in terms of their,

322
00:12:43.335 --> 00:12:44.645
their citation averages.

323
00:12:45.265 --> 00:12:48.685
Um, and maybe you could get down to, you know, 60 or 70

324
00:12:48.985 --> 00:12:51.685
or 50, but you know, for most of the schools in the middle,

325
00:12:51.685 --> 00:12:52.765
there wouldn't be a huge difference.

326
00:12:52.765 --> 00:12:53.965
Maybe at the school's at the top,

327
00:12:54.585 --> 00:12:56.445
you might see a little bit more differentiation.

328
00:12:57.585 --> 00:12:58.805
Um, and if you combined it

329
00:12:58.805 --> 00:13:01.805
with a broader index like US News, where you're looking at

330
00:13:02.525 --> 00:13:05.365
academic performance as well as incoming student credentials

331
00:13:05.385 --> 00:13:08.445
and job placement outcomes, it could matter.

332
00:13:08.545 --> 00:13:12.525
Now, I would love to see us news actually

333
00:13:13.515 --> 00:13:15.125
give us some sort of significance testing.

334
00:13:16.065 --> 00:13:17.245
The school is ranked number seven,

335
00:13:17.265 --> 00:13:20.685
but it's completely indistinguishable between schools five

336
00:13:20.685 --> 00:13:21.685
through 12, right?

337
00:13:21.685 --> 00:13:22.725
They'll never do that. Never

338
00:13:22.725 --> 00:13:23.845
in a million years will they do that.

339
00:13:23.865 --> 00:13:26.525
But I would like to see that, you know, students need

340
00:13:26.525 --> 00:13:29.045
to know that there's not really a big difference

341
00:13:29.045 --> 00:13:30.805
between the number seven school and the number eight school.

342
00:13:30.805 --> 00:13:33.645
And you shouldn't fly across the country just to move up one

343
00:13:33.645 --> 00:13:35.205
of the rankings or something like that.

344
00:13:35.785 --> 00:13:36.885
Um, you know, because students

345
00:13:36.935 --> 00:13:38.285
don't know how much weight to give this.

346
00:13:38.785 --> 00:13:40.965
Um, so I hope at least I'll get people to think about that

347
00:13:41.065 --> 00:13:42.805
and think more carefully about when differences are

348
00:13:42.805 --> 00:13:43.805
meaningful and when they're not.

349
00:13:44.065 --> 00:13:45.925
That's an interesting question.

350
00:13:45.945 --> 00:13:47.525
So there's, there's a couple databases.

351
00:13:47.545 --> 00:13:49.885
You know, Hy Online has the advantage.

352
00:13:50.705 --> 00:13:53.085
Um, they have citations for every single article.

353
00:13:53.235 --> 00:13:54.685
They actually have a spreadsheet

354
00:13:55.235 --> 00:13:57.445
with every professor's citation counts

355
00:13:57.785 --> 00:13:59.005
in the high online database.

356
00:13:59.585 --> 00:14:01.245
Um, so I'm really appreciative of that.

357
00:14:01.365 --> 00:14:02.725
I could not have done this project if I

358
00:14:02.725 --> 00:14:04.005
didn't have this data available.

359
00:14:04.865 --> 00:14:08.125
Um, but Find Online is also a very narrow database.

360
00:14:08.475 --> 00:14:11.845
It's really mostly legal, publications,

361
00:14:12.385 --> 00:14:14.765
and it omits a lot of interdisciplinary work.

362
00:14:14.765 --> 00:14:16.645
It omits books. Um,

363
00:14:16.945 --> 00:14:18.405
and so there are a lot

364
00:14:18.405 --> 00:14:20.125
of professors who've had enormous impact

365
00:14:20.125 --> 00:14:21.525
who just don't publish in law reviews,

366
00:14:22.465 --> 00:14:24.565
who look like they have no citations.

367
00:14:25.225 --> 00:14:28.205
Um, you know, people with, with incredibly prestigious

368
00:14:28.205 --> 00:14:29.565
prizes, who've written tons of books,

369
00:14:29.945 --> 00:14:31.365
but they're just completely, they look like

370
00:14:31.495 --> 00:14:32.525
zeros in high online.

371
00:14:32.525 --> 00:14:34.365
And so that's a mistake. Um,

372
00:14:34.715 --> 00:14:36.085
Westlaw does a bit of a better job.

373
00:14:36.085 --> 00:14:37.485
It's a little bit more expansive.

374
00:14:38.425 --> 00:14:40.445
Um, and because without,

375
00:14:40.445 --> 00:14:42.725
but Westlaw doesn't itself count citations.

376
00:14:43.105 --> 00:14:46.765
You have to search for somebody's name in an article.

377
00:14:46.945 --> 00:14:49.845
And that can be difficult if somebody's name is John Smith

378
00:14:49.845 --> 00:14:52.725
or David Miller or something very common.

379
00:14:53.585 --> 00:14:56.045
Um, you know,

380
00:14:56.045 --> 00:14:58.885
and then sometimes there's sort of the at all problem

381
00:14:58.995 --> 00:15:00.845
that if there's too many, you know,

382
00:15:00.845 --> 00:15:02.125
if there's too many coauthors

383
00:15:02.125 --> 00:15:03.845
and your last name starts with Z, you never,

384
00:15:03.955 --> 00:15:07.805
your name doesn't actually appear, in, in the article.

385
00:15:07.905 --> 00:15:10.765
And then the other thing is Westlaw,

386
00:15:10.875 --> 00:15:13.045
just says if your name appears in the article or not.

387
00:15:13.625 --> 00:15:15.325
So they could cite 10 of your articles,

388
00:15:15.345 --> 00:15:16.805
but Westlaw just counts at one time

389
00:15:16.805 --> 00:15:19.085
because your name appears in that article.

390
00:15:19.545 --> 00:15:20.685
And, and that may be okay.

391
00:15:20.685 --> 00:15:23.405
It may, you know, it may still be a useful measure.

392
00:15:23.865 --> 00:15:25.885
Who knows, maybe it's better to count how many,

393
00:15:25.985 --> 00:15:28.085
how many articles your name appears in rather than

394
00:15:28.085 --> 00:15:29.165
how many times you're cited.

395
00:15:29.165 --> 00:15:30.765
I, I don't really know. Um,

396
00:15:30.765 --> 00:15:32.365
but westlaw's a lot harder to work with,

397
00:15:32.365 --> 00:15:33.965
even though it seems a little bit more expansive.

398
00:15:34.065 --> 00:15:35.565
And you can also look in Google,

399
00:15:35.785 --> 00:15:39.645
but Google also has a lot of, it captures a lot

400
00:15:39.645 --> 00:15:41.845
of garbage, a lot of things that aren't really scholarship.

401
00:15:42.545 --> 00:15:45.005
Um, you know, so, so every database

402
00:15:46.265 --> 00:15:48.325
has its advantages and its problems.

403
00:15:48.985 --> 00:15:50.765
Um, now ultimately, you know,

404
00:15:50.845 --> 00:15:52.845
I don't think we can really rely on theory

405
00:15:53.345 --> 00:15:54.845
to tell us which database to use.

406
00:15:55.305 --> 00:15:56.525
You know, ultimately the question is

407
00:15:57.035 --> 00:15:58.605
each database gives its own measure

408
00:15:58.665 --> 00:16:00.485
and what corresponds to what we want.

409
00:16:01.305 --> 00:16:04.485
Um, again, with the caveat that we don't agree necessarily

410
00:16:05.105 --> 00:16:06.485
on who the best professors are,

411
00:16:06.585 --> 00:16:09.165
but you know, we can at least generate measures

412
00:16:09.165 --> 00:16:10.845
with the different databases

413
00:16:10.945 --> 00:16:12.645
or with different kinds of adjustments

414
00:16:12.665 --> 00:16:14.165
for the biases of each database.

415
00:16:14.985 --> 00:16:17.445
And, you know, measure how closely they match

416
00:16:17.445 --> 00:16:20.285
with people's perceptions of scholarly merit.

417
00:16:20.705 --> 00:16:22.445
Um, so that can help somewhat.

418
00:16:22.445 --> 00:16:25.085
But, you know, there is a, a database problem I think in

419
00:16:25.085 --> 00:16:26.365
coming years that will get easier.

420
00:16:26.605 --> 00:16:28.005
I think the databases will get better.

421
00:16:28.505 --> 00:16:31.685
Um, and I think Hein wanted its database

422
00:16:31.685 --> 00:16:32.965
to be used for citation counts.

423
00:16:32.965 --> 00:16:35.125
So they made that easy. Maybe Westlaw will

424
00:16:35.125 --> 00:16:36.205
do the same thing at some point.

425
00:16:37.025 --> 00:16:39.685
Um, but as I said, there's not really any theory.

426
00:16:39.685 --> 00:16:41.685
It's really just a matter of what matches better.

427
00:16:42.025 --> 00:16:43.485
And that's a very difficult question.

428
00:16:43.965 --> 00:16:45.485
I think one of the most important lessons is

429
00:16:45.485 --> 00:16:46.725
that findings may translate,

430
00:16:46.725 --> 00:16:49.015
but they don't translate automatically, right?

431
00:16:49.095 --> 00:16:50.495
I mean, part of my argument is

432
00:16:50.495 --> 00:16:54.095
that people in law were using methods developed in physics

433
00:16:54.235 --> 00:16:55.815
or computer science without asking

434
00:16:56.275 --> 00:16:57.295
are they applicable to law?

435
00:16:57.295 --> 00:16:59.175
That each discipline has its own practices.

436
00:16:59.715 --> 00:17:03.175
You know, as I mentioned, law reviews have hundreds,

437
00:17:03.455 --> 00:17:05.855
thousands of citations, not all of which are meaningful.

438
00:17:06.235 --> 00:17:08.295
And so we need to think about,

439
00:17:08.295 --> 00:17:10.375
whether the methods in other disciplines are actually

440
00:17:10.375 --> 00:17:12.455
appropriate for counting legal citations.

441
00:17:13.395 --> 00:17:16.215
Uh, so anyone in other field who wants

442
00:17:16.215 --> 00:17:19.055
to use my insights should just ask at least do they seem

443
00:17:19.055 --> 00:17:20.295
to apply in their own field?

444
00:17:20.435 --> 00:17:22.655
So, to give some examples, there were other studies

445
00:17:22.725 --> 00:17:24.015
that had shown before mine

446
00:17:24.015 --> 00:17:27.095
that citation counts follow log normal distribution.

447
00:17:27.235 --> 00:17:28.735
You know, that's a,

448
00:17:28.855 --> 00:17:30.895
I don't think it's a necessarily a universal finding,

449
00:17:30.915 --> 00:17:33.415
but it was known, you know, before me.

450
00:17:33.755 --> 00:17:36.055
And so in those fields, I would argue, once again,

451
00:17:36.055 --> 00:17:37.415
if you want to compare faculties,

452
00:17:37.415 --> 00:17:38.855
you should use the geometric mean.

453
00:17:39.395 --> 00:17:41.975
Um, you know, the same argument I made here,

454
00:17:42.135 --> 00:17:43.695
I think would apply to any field with

455
00:17:43.695 --> 00:17:45.095
that same kind of a distribution.

456
00:17:46.035 --> 00:17:47.535
Um, I think everyone should remember

457
00:17:47.535 --> 00:17:48.775
that citations are noisy

458
00:17:48.955 --> 00:17:50.765
Signals of scholarly impact.

459
00:17:51.425 --> 00:17:53.005
Um, and ultimately we're asking,

460
00:17:53.065 --> 00:17:54.765
is one faculty stronger than another?

461
00:17:54.905 --> 00:17:56.205
Or can we infer

462
00:17:56.205 --> 00:17:58.285
that one faculty is significantly stronger than another?

463
00:17:58.745 --> 00:18:00.485
And so we should be thinking about statistical methods

464
00:18:00.585 --> 00:18:02.525
to determine whether differences are meaningful or not.

465
00:18:02.945 --> 00:18:04.405
And we have to remember what are these rankings

466
00:18:04.405 --> 00:18:05.685
good for, right?

467
00:18:05.685 --> 00:18:08.445
And ultimately, they're giving guidance to people, you know,

468
00:18:08.445 --> 00:18:10.645
prospective students, maybe prospective faculty

469
00:18:10.855 --> 00:18:11.925
about where they should go.

470
00:18:12.425 --> 00:18:13.605
And so we really need to know,

471
00:18:13.625 --> 00:18:15.045
and not just, you know,

472
00:18:15.175 --> 00:18:16.645
which school ranks higher than the other,

473
00:18:16.645 --> 00:18:18.205
but are these differences actually meaningful?

474
00:18:18.905 --> 00:18:21.245
Um, you know, I think the statistical tests I use

475
00:18:21.245 --> 00:18:22.325
would certainly be applicable.

476
00:18:22.325 --> 00:18:24.965
Again, if you have a long normal distribution, you'd have

477
00:18:24.965 --> 00:18:27.085
to modify them a little bit if the citations are

478
00:18:27.085 --> 00:18:28.365
distributed some other way.

479
00:18:29.025 --> 00:18:32.325
Um, and so I certainly hope the methods I developed can be

480
00:18:32.325 --> 00:18:33.485
applied in other fields or

481
00:18:33.485 --> 00:18:35.205
that people will improve on them in various ways.

482
00:18:35.825 --> 00:18:37.645
And, and who knows, maybe I'll even get some citations

483
00:18:37.645 --> 00:18:37.845
for it.

Evaluating the Significance of Law School Citation Rankings

00:00:00 - A general issue with law school citation rankings is law schools. Many law schools are very closely clustered together. And anyone in the past who had ever published law school citation citation rankings is acknowledged. small differences don't really matter. but there were no criteria for determining whether differences are meaningful or not. It was really all just completely arbitrary approaches to determining whether differences mattered or not. Whereas in most of legal scholarship, we actually look to see do, or empirical scholarship,

00:00:34 - Are differences significant or estimates significant? can you actually look from the data and say that this tells us that there's some underlying measure that is actually distinct between say, Harvard Law School and some other law school? We should think about the citations not as the object of scholarship, right? Scholarship is not a game where whoever gets the most citations wins, you know, scholarship. You're trying to actually advance knowledge. And citations are a noisy measure of, you know,

00:01:07 - And so I was thinking more about the data generating process. You know, how do you know faculties, hire professors, professors write articles. Those articles get cited. and that gives us a signal about the quality, the scholarly quality of the professor, and the qual the scholarly quality of the faculty. And then can we look at those signals and think about the data generating process and the statistical distribution and ask, well, are these differences really significant?

Evaluating Faculty Quality Through Citation Analysis

00:01:39 - of a professor on one faculty is actually higher than the average quality another faculty? And no one had really tried to do that before. so I found that the citations were logged, normally distributed, and that was an important part of, of this project. and in part because the log normal distribution is extremely skewed. so superstar professors look like they're so much more important than everyone else.

00:02:08 - other fields that, citations kind of accumulate through an exponential process, right? You, you publish an article, it gets cited. And when it gets cited, that means more people discover your work. And the more people discover your work, the more they cite it. and then it, I said, it grows exponentially, and then it also enhances your reputation. So when you publish new articles, people immediately wanna read them because they know your reputation. And so citations beget citations, and citation counts grow exponentially.

00:02:38 - for some scholars than for others based on the contributions in their work. but ultimately we need to think about how this relates to what we think of as equality. And I do think that citations are in some sense, exponential for scholar equality. And then the log normal distribution helps us both because it gives us a statistical test for comparing the two faculties. We can ask, does it look like their citations are drawn from the same distribution, or does one faculty's distribution look like it dominates another?

Evaluating Citation Rankings and Student Priorities

00:03:11 - for essentially averaging the citations within each faculty. and I think that's important because, people have been using very arbitrary measures. Haven't thought about why, given that the differences in the citation counts are mostly insignificant for any kind of faculties that are thought of as peers or comparable. I'll be honest, I don't think prospective students or should be giving any weight at all

00:03:40 - Now, now, I'm gonna be honest as well. faculty may care. I don't think most prospective students really care that much about how many citations their perspective, professors have. but they do give an enormous amount of weight to the rankings that are published by US News. now those are focused more on employment outcomes. Do graduates pass the bar and get good jobs to some degree incoming students PR credentials? Will my peers, have good credentials?

00:04:11 - outside the rankings. Are the students happy? Do professors care about teaching? That sort of thing. so I think citation rankings have been something that faculty are more fixated on than students right now. but they do matter for very important reason. And that is that there's speculation that they could be incorporated into general rankings like the US News rankings. US news even announced several years ago that they were gonna publish a citation ranking of law schools. They abandoned that plan, at least the time being in part

The Challenges of Citation Rankings in Law School Assessments

00:04:42 - It's not that easy thing to do, but there's certainly a potential it could be revived. US News has historically, measured the scholarly reputation of faculties using surveys, but that's been widely criticized. The average professor really isn't following that much what's going on at other schools. and so it seems to just sort of perpetuate the hierarchy. And so a lot of people have proposed citation rankings as an alternative. but the problem is, if the differences among faculties are mostly statistical noise,

00:05:12 - You know, first it can lead to misleading results. The rankings could amplify what are really meaningless differences. but the second problem is that law schools put a lot of effort into gaming the rankings. students care about the rankings and law schools need to make sure that they're as highly ranked as they can. Deans get fired because they, they drop one or two places in the rankings. and so they could also try to game citation rankings. you can any law school can,

00:05:43 - by hiring faculty in widely cited fields like constitutional law, avoid tax or specialty fields, or interdisciplinary work fields are cited less. they could hire more senior professors who have lots of citations and avoid hiring young people. now that could be good for school's rankings, but that's not good for the academy. It's not good for scholarship. It's not good for teaching. and so I wanna be clear if, if we're really talking about incorporating this into a broader ranking, this could lead to a lot of gaming

Evaluating Citation Counts as Predictors of Career Advancement

00:06:13 - 'cause it doesn't seem like it's very informative. I have a another paper with my colleague Michael Livermore, where we're looking at how well citation ranking, or I'm sorry, citation counts, predict career advancements. So for example, do professors move, get jobs at more prestigious law schools? Professors who are highly cited. And we find that it is has some predictive value, but you get a lot more predictive value if you look at other measures besides citation. So for example,

00:06:44 - Do they publish in prestigious journals? do they, author books that are published by prestigious university presses? So if you incorporate something that was a little broader, we find that actually predicts career advancement better. Now, not everyone agrees with looking at career advancement. You know, not everyone agrees with the choices that Harvard, Yale and Stanford make in terms of who they hire. but you do find that those top schools hire a lot of people who don't have high citation counts. They might be legal historians,

00:07:15 - and new fields that are emerging where there aren't a lot of citations yet. and so there could be a, a bit of a bias. and in legal scholarship in particular, you know, large articles could have 500 footnotes with thousands of citations. And not every one of them is really meaningful. It's not like physics where you have a few dozen citations and you're really giving intellectual credit, to the other authors. Some of them are not that meaningful. And sometimes people get

Evaluating Citation Measures in Scholarship

00:07:45 - And sometimes people get cited because they wrote a well-written survey article. that is the go-to article. If you're writing about something in particular, I would be a little concerned if we just blindly counted citations without thinking about how they correspond to the way we perceive merit with, with the caveat that we don't agree on what merit is. Some people might really like survey articles more than they like interdisciplinary work. You know, I, I don't, but that's, you know, there are some, some disagreements there. But I do think we have to be careful and there's lots of ways to count citations

00:08:18 - will give us different results. And we have to think carefully. If we incorporate anything into the rankings, how would we do it? If you are comparing the citations on two different faculties, you need some measure to summarize the citations on each faculty. Now, a pioneering study of law school citation counts that actually had a sort of rudimentary ranking from 25 years ago, thought about this question. They said, you know, you could use the mean, you could use the median. they couldn't decide which is better.

00:08:49 - and the median as a kind of throwing their hands up in the air, what measures should I use? And then, later on, another ranking by, a professor who has a very, prominent online presence decided to use twice the mean plus the median. 'cause I think, well, I don't really know why of it was still pretty arbitrary. And then others started to follow that. And, you know, there's studies that actually would say, this measure doesn't really make any sense, but everyone else is using it. And we wanna be consistent with the

Evaluating Law School Rankings: The Role of Geometric Mean

00:09:19 - and that's a terrible way to rank law schools. And of course, the rankings look different depending on which measure you use. and if you can't give a reason for choosing one me measure over another, then you really can't give a reason for choosing one ranking over another. that necessarily means there'll be a lot of arbitrariness. So, my choice of geometric mean was motivated by the distribution of the citations. The fact that you have, log normal distribution.

00:09:52 - measure of central tendency for the log normal distribution. So it's essentially sort of squeezing the most information out of the data, gives us the best estimate of central tendency. Also gives us very easy statistical tests to see if the citations from two faculties were drawn from the same distribution or not. now another practical effect of the geometric mean is that it's much less sensitive to the outliers. as I mentioned, it's a very skewed distribution.

00:10:21 - it really is enormously influenced by the presses at the very top of the distribution. As I mentioned, you know, the citations accumulate through an exponential process. And so that means that the very top professors really dominate. And so, lemme give you an example. the most cited professor in my data was Professor Casson Stein at Harvard Law School. in the last five years, he had 5,800 citations a little bit more. and that's about 24 times as many citations as the median faculty member at Harvard.

Evaluating Law Faculty: Beyond Superstars and Citations

00:10:52 - I have enormous respect for him. but I would be a little skeptical of any measure that values him 24 times as much as a typical Harvard Law School, faculty member. and so if you're using a measure that's based on the arithmetic meme, you're really looking at how many cast sunscreens does each faculty have. and maybe there's some people who care about that. They only wanna know how many superstars are there on a particular faculty. Although, once again, you have to both a superstar and in the highly sighted field, attack Superstar is still gonna look pretty adorable.

00:11:26 - about the entire faculty, not just the most famous. If you're a student, you're not gonna take all your classes with Cass Sunstein. and so the geometric mean, I think better captures the whole range of members on each faculty. So that's sort of another intuitive advantage. I think the wide confidence, and it'll show that these rankings don't really convey a lot of useful information. but it is possible that we could devise other measures that are more useful.

00:11:57 - to predict whether, you know, people get jobs at better law schools, looking at where they publish, looking at books, there might be other kinds of scholarly indicia that would be relevant. maybe awards, you know, you'd have to think about it, but it's possible if you combine citations with other measures, you might get something a little bit more informative. And the same is true, I think, if you account for seniority. but I'm a little skeptical

Challenges in Assessing Law School Citation Averages

00:12:28 - These contents intervals were so wide. I mean, shockingly wide. I had no idea when I started this project, I was gonna get, you know, the schools in the middle. You basically can't tell them apart. There's a a hundred some schools that are more or less indistinguishable, in terms of their, their citation averages. and maybe you could get down to, you know, 60 or 70 or 50, but you know, for most of the schools in the middle, there wouldn't be a huge difference. Maybe at the school's at the top, you might see a little bit more differentiation.

00:12:58 - with a broader index like US News, where you're looking at academic performance as well as incoming student credentials and job placement outcomes, it could matter. Now, I would love to see us news actually give us some sort of significance testing. The school is ranked number seven, but it's completely indistinguishable between schools five through 12, right? They'll never do that. Never in a million years will they do that. But I would like to see that, you know, students need to know that there's not really a big difference

00:13:30 - And you shouldn't fly across the country just to move up one of the rankings or something like that. you know, because students don't know how much weight to give this. so I hope at least I'll get people to think about that and think more carefully about when differences are meaningful and when they're not. That's an interesting question. So there's, there's a couple databases. You know, Hy Online has the advantage. they have citations for every single article. They actually have a spreadsheet with every professor's citation counts in the high online database.

Challenges of Legal Citation Databases

00:14:01 - I could not have done this project if I didn't have this data available. but Find Online is also a very narrow database. It's really mostly legal, publications, and it omits a lot of interdisciplinary work. It omits books. and so there are a lot of professors who've had enormous impact who just don't publish in law reviews, who look like they have no citations. you know, people with, with incredibly prestigious prizes, who've written tons of books,

00:14:31 - zeros in high online. And so that's a mistake. Westlaw does a bit of a better job. It's a little bit more expansive. and because without, but Westlaw doesn't itself count citations. You have to search for somebody's name in an article. And that can be difficult if somebody's name is John Smith or David Miller or something very common. you know, and then sometimes there's sort of the at all problem that if there's too many, you know,

00:15:02 - and your last name starts with Z, you never, your name doesn't actually appear, in, in the article. And then the other thing is Westlaw, just says if your name appears in the article or not. So they could cite 10 of your articles, but Westlaw just counts at one time because your name appears in that article. And, and that may be okay. It may, you know, it may still be a useful measure. Who knows, maybe it's better to count how many, how many articles your name appears in rather than how many times you're cited. I, I don't really know. but westlaw's a lot harder to work with,

Evaluating Legal Citation Databases: Strengths and Challenges

00:15:34 - And you can also look in Google, but Google also has a lot of, it captures a lot of garbage, a lot of things that aren't really scholarship. you know, so, so every database has its advantages and its problems. now ultimately, you know, I don't think we can really rely on theory to tell us which database to use. You know, ultimately the question is each database gives its own measure and what corresponds to what we want. again, with the caveat that we don't agree necessarily

00:16:06 - but you know, we can at least generate measures with the different databases or with different kinds of adjustments for the biases of each database. And, you know, measure how closely they match with people's perceptions of scholarly merit. so that can help somewhat. But, you know, there is a, a database problem I think in coming years that will get easier. I think the databases will get better. and I think Hein wanted its database to be used for citation counts. So they made that easy. Maybe Westlaw will

00:16:37 - but as I said, there's not really any theory. It's really just a matter of what matches better. And that's a very difficult question. I think one of the most important lessons is that findings may translate, but they don't translate automatically, right? I mean, part of my argument is that people in law were using methods developed in physics or computer science without asking are they applicable to law? That each discipline has its own practices. You know, as I mentioned, law reviews have hundreds, thousands of citations, not all of which are meaningful.

Evaluating Legal Citation Methods Across Disciplines

00:17:08 - whether the methods in other disciplines are actually appropriate for counting legal citations. so anyone in other field who wants to use my insights should just ask at least do they seem to apply in their own field? So, to give some examples, there were other studies that had shown before mine that citation counts follow log normal distribution. You know, that's a, I don't think it's a necessarily a universal finding, but it was known, you know, before me. And so in those fields, I would argue, once again, if you want to compare faculties,

00:17:39 - you know, the same argument I made here, I think would apply to any field with that same kind of a distribution. I think everyone should remember that citations are noisy Signals of scholarly impact. and ultimately we're asking, is one faculty stronger than another? Or can we infer that one faculty is significantly stronger than another? And so we should be thinking about statistical methods to determine whether differences are meaningful or not. And we have to remember what are these rankings good for, right? And ultimately, they're giving guidance to people, you know,

00:18:37 - about where they should go. And so we really need to know, and not just, you know, which school ranks higher than the other, but are these differences actually meaningful? you know, I think the statistical tests I use would certainly be applicable. Again, if you have a long normal distribution, you'd have to modify them a little bit if the citations are distributed some other way. and so I certainly hope the methods I developed can be applied in other fields or that people will improve on them in various ways. And, and who knows, maybe I'll even get some citations for it.

Faculti AI can make mistakes

Research

Policy

Analysis