Popular in Professional

Architecture

Connecting Cities and Nature

University of Virginia Timothy Beatley

Human Rights Law

Human Rights and Corruption: Problems and Potential of Individualising a Systemic Problem

Max Planck Institute Anne Peters

Finance

Chinese Exchange Rate Policy: Lessons for Global Investors

University of California San Diego Michael Melvin

Law

A statistical approach to law school citation rankings

University of Virginia Joshua Fischman

Education

Firm Performance Pay as Insurance against Promotion Risk

Stockholm School of Economics Alvin Chen

Constitutional Law

What is constitutional ideology?

University of Oxford Nick Barber

The Narrative AI Advantage?

View more insights in Professional - Business - Information Systems

View more figures from University of Washington

View the page of Leonard Boussioux

Read the Study

Next in Business

Feature Misspecification in Sequential Learning Problems

Dongwook Shin | Hong Kong University of Science and Technology

The Allegory of the Favela: The Multifaceted Effects of Socioeconomic Mobility

Leandro S. Pongeluppe | University of Pennsylvania

How can human-AI collaboration transform the evaluation of creative solutions, and what role does critical thinking play in balancing AI’s influence on both objective and subjective criteria in decision-making processes?

Image courtesy of interviewee. October 16, 2024

Log-in or Sign-up to Faculti
Currently viewing this subject insight as a guest. You have insight(s) remaining for this month. Login to view 8000+ figures on the platform.

Generating Workspace...

Export as PDF

This video video discusses research on the future of work, particularly how artificial intelligence (AI) can enhance problem-solving and innovation. The author emphasizes the importance of human-AI collaboration, particularly in evaluating ideas and solutions to complex challenges, such as the United Nations' sustainable development goals. The research involves working with MIT Solve, which funds early-stage projects and faces the challenge of evaluating thousands of applications.

The author describes a study aimed at improving the idea evaluation process using generative AI, specifically GPT-4. The AI is designed to assist human evaluators by providing recommendations and explanations for its decisions. Results from experiments reveal that when evaluators incorporate AI, they often become more discerning in their judgments.

Key insights from the study include:
1. AI can help standardize and expedite the evaluation of objective criteria, improving the efficiency of decision-making.
2. Evaluators are influenced by AI recommendations, with both novice and expert evaluators reacting similarly to AI inputs.
3. Human oversight remains essential for subjective evaluations, allowing for a balance of augmentation rather than replacement by AI.
4. Effective AI-human collaboration requires understanding how to operate and engage with AI tools to maximize their potential.

Overall, the study underscores the need for thoughtful design in human-AI partnerships to ensure that the benefits of AI are fully realized while maintaining the critical role of human insight and creativity in the decision-making process.

Language:

Export as PDF

Export as PDF
Here are five key points from the video:

1. **Impact of AI on Problem Solving**: The author is researching how artificial intelligence (AI) can enhance innovation and problem-solving processes, particularly in addressing significant global challenges like the United Nations' Sustainable Development Goals (SDGs).

2. **Human-AI Collaboration**: The paper emphasizes the importance of collaborating with AI rather than replacing human input. It explores how AI can augment human decision-making and creativity by assisting in the evaluation and selection of innovative ideas.

3. **Evaluation of Ideas**: The research aims to streamline the idea evaluation process for organizations like MIT Solve, which receives thousands of applications for funding. AI tools can help sift through these applications and provide recommendations, ultimately leading to a more efficient selection of promising projects.

4. **Role of Expertise and Training**: The interaction between human evaluators and AI is highlighted, noting that individuals who engage more deeply with both the AI recommendations and the evaluated solutions tend to make more informed decisions, demonstrating the significance of upskilling users in understanding AI tools.

5. **Subjectivity vs. Objectivity in Evaluation**: The study finds that the effectiveness of AI in assisting human evaluators varies depending on whether the criteria for evaluation are objective or subjective. For objective criteria, AI can automate evaluations effectively; however, for subjective assessments, human input remains crucial to ensure fairness and to leverage personal intuition.

Export as PDF
Here are some key quotes gathered from your video, highlighting important themes and insights regarding the intersection of technology, particularly AI, and the future of work:

1. **Impact of AI on Innovation**:
- "My goal is to accelerate problem solving... I'm focusing these days on multiple stages of how can we solve big problems like the sustainable development goals."
- "One part of solving problems is to find great ideas... You need to go from a spark of an idea to an actual prototype."

2. **Human-AI Collaboration**:
- "The whole point is the human AI collaboration. We don't want to replace humans. We want to augment our decision making."
- "I believe that we won't be able to remove the human in the loop."

3. **AI’s Role in Decision-Making**:
- "Whenever you have the AI, you tend to become harsher... We realized that naturally people are kinder than they should be."
- "We trained our AI to reflect how the professional solvers were screening out in the previous years."

4. **Literature on Human-AI Interaction**:
- "There is a literature showing that collaborating with AI can make you obviously better... However, in certain situations, it can also make you worse."
- "It's absolutely not guaranteed to have a situation like this."

5. **Creative Potential of AI**:
- "I'm a huge advocate of human creativity... using generative AI feels like playing with a tool like a brush."
- "I want people to feel like artists again."

6. **Evaluation of Ideas at Scale**:
- "They want to go from those thousands of solutions to a subset of maybe a hundred or 200 that they shortlisted."
- "We wanted to replicate this entire pipeline with large language models."

7. **Insights on Expertise and Bias**:
- "We realize that if you want to evaluate something subjective... people will feel more convinced than if they just have the AI recommendation."
- "We confirmed that even the experts were influenced the same way as non-experts."

8. **Automation and Subjectivity**:
- "It looks like it's easier to automate... Whenever it involves subjectivity, we don't want to remove the humans."
- "AI will read everything... It brings more fairness somehow if you properly calibrate."

These quotes capture the essence of the discussion on how AI can enhance human innovation and decision-making processes while also examining the potential challenges and implications of integrating AI tools in the workplace.

X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:90000

1
00:00:00.185 --> 00:00:01.765
I'm researching the future of work.

2
00:00:02.125 --> 00:00:05.045
I want to know how technologies such as

3
00:00:05.605 --> 00:00:07.885
artificial intelligence will impact the way we work.

4
00:00:08.825 --> 00:00:10.845
My goal is to accelerate problem solving.

5
00:00:11.605 --> 00:00:14.325
I want to help more people to achieve more things.

6
00:00:15.445 --> 00:00:18.185
And in particular, I'm focusing these days

7
00:00:18.765 --> 00:00:20.505
on multiple stages of

8
00:00:20.605 --> 00:00:23.065
how can we solve big problems like the sustainable

9
00:00:23.065 --> 00:00:24.865
development goals by the United Nations.

10
00:00:25.885 --> 00:00:29.465
One part of solving problems is to find great ideas

11
00:00:29.605 --> 00:00:31.345
to solving challenging problems,

12
00:00:32.045 --> 00:00:34.585
but then you need to develop those ideas.

13
00:00:34.645 --> 00:00:37.105
You need to go from a spark of an idea

14
00:00:37.105 --> 00:00:41.945
to an actual prototype, realiz realization, pro product.

15
00:00:42.865 --> 00:00:44.545
I want to build AI tools

16
00:00:45.005 --> 00:00:49.105
and tech technological tools to help people in every stage.

17
00:00:50.155 --> 00:00:52.495
The research that we're going to talk about today is

18
00:00:52.495 --> 00:00:54.695
around the early stages of innovation.

19
00:00:55.195 --> 00:00:56.655
So a previous study that we've done

20
00:00:56.855 --> 00:00:59.335
with my colleagues at Harvard is around

21
00:00:59.595 --> 00:01:02.215
how can we leverage generative AI

22
00:01:02.715 --> 00:01:04.335
to help us have better ideas?

23
00:01:04.675 --> 00:01:08.205
And the whole process of human AI collaboration

24
00:01:08.345 --> 00:01:10.485
for finding ideas to challenging problems.

25
00:01:10.945 --> 00:01:12.685
The project we are going to talk about today

26
00:01:13.345 --> 00:01:17.485
is extremely important for any company, organization,

27
00:01:18.405 --> 00:01:21.645
students, learners who are facing many, many choices,

28
00:01:22.225 --> 00:01:25.525
and they wish to have the help of a tool like AI

29
00:01:25.945 --> 00:01:28.565
to help them select the best ideas, the ideas

30
00:01:28.565 --> 00:01:31.965
that they should push more, interest in.

31
00:01:31.965 --> 00:01:33.605
They're collaborating with MIT Solve.

32
00:01:33.755 --> 00:01:35.445
It's a marketplace for innovation.

33
00:01:35.555 --> 00:01:37.165
They give about 70,

34
00:01:37.435 --> 00:01:42.405
they've been giving $70 million in funding to so many,

35
00:01:42.495 --> 00:01:46.675
early stage projects all over the world to help,

36
00:01:47.145 --> 00:01:51.595
them solve things around financial inclusion, global health,

37
00:01:52.415 --> 00:01:54.235
education for all.

38
00:01:55.185 --> 00:01:58.085
And they receive thousands of applications

39
00:01:58.145 --> 00:02:00.325
to their crowdsourcing challenges.

40
00:02:00.325 --> 00:02:02.645
So their goal is to get solutions from all over the world,

41
00:02:02.945 --> 00:02:05.085
but then they have thousands of people to evaluate,

42
00:02:05.385 --> 00:02:09.045
and every solution to evaluate can be from five to 15 pages.

43
00:02:09.875 --> 00:02:12.385
Their goal is to go from those thousands of solutions

44
00:02:12.845 --> 00:02:17.345
to a subset of maybe a hundred or 200 that they shortlisted,

45
00:02:17.345 --> 00:02:19.505
and then they want to find which ones should be the

46
00:02:19.505 --> 00:02:21.505
finalists and which one should be the winners.

47
00:02:21.775 --> 00:02:23.585
It's a super time consuming process

48
00:02:24.165 --> 00:02:25.745
and it's very overwhelming to be able

49
00:02:25.745 --> 00:02:27.625
to ski through so much content.

50
00:02:27.725 --> 00:02:30.705
So the whole point of this paper is to build

51
00:02:31.465 --> 00:02:34.505
a human AI collaboration to help us

52
00:02:35.705 --> 00:02:36.985
evaluate ideas at scale.

53
00:02:37.445 --> 00:02:39.585
The whole point is the human AI collaboration.

54
00:02:39.645 --> 00:02:40.865
We don't want to replace humans.

55
00:02:41.165 --> 00:02:43.625
We want to augment our decision making.

56
00:02:43.655 --> 00:02:45.665
That meaning have the people who will listen

57
00:02:45.665 --> 00:02:48.705
to these have used chat pt, maybe they created images,

58
00:02:48.845 --> 00:02:50.745
for instance with Mid Journey or Dali.

59
00:02:51.255 --> 00:02:54.345
Basically, the computer science literature focuses on

60
00:02:54.345 --> 00:02:56.305
deploying better technology,

61
00:02:56.885 --> 00:02:58.865
but they don't always necessarily study

62
00:02:59.225 --> 00:03:01.845
how those technologies will impact the future

63
00:03:01.905 --> 00:03:03.285
of work or businesses.

64
00:03:03.715 --> 00:03:07.485
This is where there is another literature entering, which is

65
00:03:08.305 --> 00:03:11.005
how are humans collaborating with technologies?

66
00:03:11.105 --> 00:03:13.205
How are they using it? What's the optimal way,

67
00:03:13.785 --> 00:03:17.205
the optimal division of labor with the ai?

68
00:03:17.745 --> 00:03:20.945
So there is a literature showing that collaborating

69
00:03:20.945 --> 00:03:23.425
with AI can make you obviously better.

70
00:03:23.485 --> 00:03:25.305
It can help upskill people.

71
00:03:25.605 --> 00:03:28.785
It can help you do tasks that were deemed infeasible before.

72
00:03:29.055 --> 00:03:32.145
However, in certain situations, it can also make you worse

73
00:03:32.175 --> 00:03:35.105
because you are sort of outside of the realm

74
00:03:35.105 --> 00:03:36.185
of what AI can do.

75
00:03:36.645 --> 00:03:39.025
And then you should have relied on your own expertise versus

76
00:03:39.075 --> 00:03:40.785
using, for instance, chat g pity.

77
00:03:41.455 --> 00:03:42.835
So there is a literature on that.

78
00:03:42.845 --> 00:03:44.595
There is also a literature on studying

79
00:03:45.295 --> 00:03:47.475
in which scenarios having a human

80
00:03:47.575 --> 00:03:49.155
and an AI collaborating together

81
00:03:49.945 --> 00:03:53.275
will outperform the AI alone and the human alone.

82
00:03:53.775 --> 00:03:55.515
And it's absolutely not guaranteed

83
00:03:55.515 --> 00:03:56.595
to have a situation like this.

84
00:03:56.665 --> 00:03:58.315
It's really possible that human AI

85
00:03:58.825 --> 00:04:00.915
outperforms the human only,

86
00:04:01.335 --> 00:04:03.195
but the AI alone would fare better

87
00:04:04.055 --> 00:04:06.875
or the human would fare better than the human ai.

88
00:04:06.975 --> 00:04:09.115
So this is where there is the literature.

89
00:04:09.455 --> 00:04:11.435
And so we enter this literature to show

90
00:04:11.865 --> 00:04:15.715
what happens when we do evaluations, which is something

91
00:04:15.715 --> 00:04:18.435
that has not inve been investigated much yet.

92
00:04:18.875 --> 00:04:21.715
I am a huge advocate of human creativity.

93
00:04:22.155 --> 00:04:25.955
I love being able to figure out how to solve problems,

94
00:04:26.175 --> 00:04:29.035
but I'm also a huge advocate of leveraging technology

95
00:04:29.055 --> 00:04:30.675
to make us even more creative.

96
00:04:31.575 --> 00:04:34.235
And then what I witnessed is my students,

97
00:04:34.265 --> 00:04:36.395
like I teach generative AI to hundreds

98
00:04:36.395 --> 00:04:39.675
of students every year, to companies, to organizations.

99
00:04:40.375 --> 00:04:44.425
And I always remarked that we lack creativity in general

100
00:04:44.975 --> 00:04:47.145
because we don't think ourselves as artists

101
00:04:47.145 --> 00:04:49.505
because we don't think of extra use cases.

102
00:04:49.965 --> 00:04:53.625
So one of the key things in my teaching is I want people

103
00:04:53.725 --> 00:04:55.425
to feel like artists again,

104
00:04:55.425 --> 00:04:58.025
because I believe that using generative AI

105
00:04:58.595 --> 00:05:01.265
feels like playing with a tool like a brush.

106
00:05:02.005 --> 00:05:04.865
And I want people to feel that they can innovate

107
00:05:04.935 --> 00:05:06.945
through their way of using ai,

108
00:05:07.365 --> 00:05:08.585
and they can find new cases

109
00:05:08.615 --> 00:05:10.665
that nobody has investigated before.

110
00:05:11.125 --> 00:05:13.665
So the human AI collaboration is an opportunity

111
00:05:13.725 --> 00:05:15.145
for even more creativity,

112
00:05:15.445 --> 00:05:18.345
and I believe that this is how you can get some exponential

113
00:05:19.165 --> 00:05:21.945
growth solving because we collaborate,

114
00:05:22.005 --> 00:05:25.145
but we need the human ingenuity, the human craft

115
00:05:25.705 --> 00:05:27.145
thinking to get there.

116
00:05:27.505 --> 00:05:29.345
I believe that we won't be able

117
00:05:29.345 --> 00:05:31.065
to remove the human in the loop.

118
00:05:31.885 --> 00:05:34.745
And I, I think there are two things that can happen.

119
00:05:35.085 --> 00:05:37.465
AI in the loop, human in the loop and variations,

120
00:05:37.925 --> 00:05:42.025
but basically some AI systems, they try to remove

121
00:05:42.025 --> 00:05:43.225
as much human as possible.

122
00:05:43.285 --> 00:05:46.105
So if you look at the latest model from church PTs

123
00:05:46.205 --> 00:05:50.505
or one one, this model is supposedly thinking.

124
00:05:50.565 --> 00:05:52.785
So OpenAI says it's thinking, it's actually not thinking.

125
00:05:52.785 --> 00:05:54.385
It's more like processing information.

126
00:05:55.005 --> 00:05:56.865
And then you don't see exactly what it thinks.

127
00:05:57.105 --> 00:05:58.765
You see like a chain of thought.

128
00:05:58.785 --> 00:06:01.525
So you see like, what's the model doing right now?

129
00:06:01.745 --> 00:06:02.805
You don't see the details

130
00:06:03.225 --> 00:06:05.005
and it just gives you an output

131
00:06:05.005 --> 00:06:07.005
that is probably much better.

132
00:06:07.305 --> 00:06:09.685
Uh, if it's around math and science than many models

133
00:06:09.705 --> 00:06:11.005
before, it's very impress.

134
00:06:11.035 --> 00:06:13.005
I've very, I've been very, I impressive with some use cases.

135
00:06:13.625 --> 00:06:15.965
That's an example where they sort

136
00:06:15.965 --> 00:06:17.365
of remove the human in the middle

137
00:06:17.625 --> 00:06:20.165
and you can iterate with the final content,

138
00:06:20.265 --> 00:06:22.205
but not really with what's happening in the middle.

139
00:06:22.725 --> 00:06:25.525
I am sure that use cases like this will happen.

140
00:06:25.585 --> 00:06:28.805
So I agree that in some scenarios we are going

141
00:06:28.805 --> 00:06:31.525
to remove the human because it's really in the range of

142
00:06:31.525 --> 00:06:33.125
what AI can do very well.

143
00:06:33.505 --> 00:06:36.085
We will just need to be able to check. So that's important.

144
00:06:36.205 --> 00:06:38.325
I think that humans can have the control on what's

145
00:06:38.325 --> 00:06:40.445
happening, but then the humans will work

146
00:06:40.475 --> 00:06:41.525
with the final part.

147
00:06:42.425 --> 00:06:45.645
Yet I think we, I'm not seeing the technology

148
00:06:45.785 --> 00:06:47.805
to be good enough that we'll just have

149
00:06:47.805 --> 00:06:48.925
to focus on creativity.

150
00:06:48.935 --> 00:06:51.165
We'll have to focus on interacting as well,

151
00:06:51.945 --> 00:06:55.275
or at least there will be a decent amount of time,

152
00:06:55.275 --> 00:06:58.395
maybe at least 3, 4, 5 years if if not more.

153
00:06:58.575 --> 00:07:00.875
And I believe more where we have

154
00:07:00.895 --> 00:07:02.995
to keep collaborating all along the process.

155
00:07:03.535 --> 00:07:05.275
And I believe that this is one of the points

156
00:07:05.275 --> 00:07:06.555
that we are making in the paper, that

157
00:07:06.555 --> 00:07:07.635
we have domain expertise.

158
00:07:07.775 --> 00:07:10.475
So that's traditional knowledge of a given field.

159
00:07:10.775 --> 00:07:14.675
For instance, physics, math, computer science, innovation.

160
00:07:15.415 --> 00:07:19.075
But also we see the emergence of a new type of expertise,

161
00:07:19.135 --> 00:07:23.235
the AI interaction, expertise, your ability to interact

162
00:07:23.235 --> 00:07:25.595
with the AI to get the AI to do what you want.

163
00:07:26.015 --> 00:07:28.835
And this is not only creativity, it's also a skill.

164
00:07:28.975 --> 00:07:32.555
The ability to talk to the AI to shift,

165
00:07:32.935 --> 00:07:35.555
to nudge the ai, to do what you really wanted to do.

166
00:07:35.735 --> 00:07:37.995
Our first goal was to actually collaborate

167
00:07:37.995 --> 00:07:39.475
with a real world platform.

168
00:07:39.615 --> 00:07:42.315
We don't want to do work at theoretical.

169
00:07:43.095 --> 00:07:44.235
And even though we would do field

170
00:07:44.235 --> 00:07:45.515
experiments, we would get insights.

171
00:07:45.535 --> 00:07:47.515
We really wanted to use real world data

172
00:07:47.935 --> 00:07:51.075
and help actual companies or organizations.

173
00:07:51.095 --> 00:07:53.035
So we did this collaboration with MIT solve.

174
00:07:53.305 --> 00:07:56.555
They were amazing because they gave us an amount of data

175
00:07:56.585 --> 00:07:59.915
that is beautiful and this data is so precious.

176
00:08:00.185 --> 00:08:01.995
It's people who've been working

177
00:08:02.015 --> 00:08:03.315
to make the world a better place,

178
00:08:03.335 --> 00:08:05.195
and we have thousands of such ideas.

179
00:08:05.255 --> 00:08:08.995
So that was the first principle, a great collaboration.

180
00:08:09.975 --> 00:08:13.955
Second, we wanted to understand how they actually

181
00:08:15.035 --> 00:08:16.035
evaluate ideas at scale.

182
00:08:16.215 --> 00:08:20.715
And so what they do is that they hire, externally, they,

183
00:08:20.785 --> 00:08:22.755
they have volunteers, committee volunteers

184
00:08:22.755 --> 00:08:24.395
to help them screen solutions,

185
00:08:24.895 --> 00:08:26.755
and they also have internal judges

186
00:08:26.855 --> 00:08:29.115
and staff to go through thousands

187
00:08:29.115 --> 00:08:30.835
of solutions every time is a new challenge.

188
00:08:31.645 --> 00:08:33.625
And then they have a clean process of doing this.

189
00:08:33.625 --> 00:08:36.385
They have a hierarchy of criteria that every idea has

190
00:08:36.385 --> 00:08:38.025
to pass, and then it goes

191
00:08:38.025 --> 00:08:40.025
through multiple screeners, et cetera.

192
00:08:40.505 --> 00:08:42.965
We wanted to replicate this entire pipeline

193
00:08:43.035 --> 00:08:44.205
with large language models.

194
00:08:44.545 --> 00:08:46.285
In particular, we use GPT-4,

195
00:08:46.875 --> 00:08:49.725
some sophisticated prompt engineering to be able

196
00:08:49.725 --> 00:08:52.085
to tell the model how to evaluate an idea.

197
00:08:52.665 --> 00:08:55.525
And thanks to this we collected for every idea.

198
00:08:56.535 --> 00:08:58.555
So it's a solution that can be from T five

199
00:08:58.615 --> 00:09:02.115
to 15 pages about what's your solution about who's working

200
00:09:02.175 --> 00:09:06.035
on it, and are you at a prototype stage,

201
00:09:06.425 --> 00:09:09.675
extract the, the technology behind, et cetera.

202
00:09:09.855 --> 00:09:11.515
We have an AI that can read all that

203
00:09:11.575 --> 00:09:14.755
and tell us if it thinks the solution will pass

204
00:09:14.775 --> 00:09:16.435
or fail every criterion.

205
00:09:17.365 --> 00:09:19.185
So now we have AI recommendations.

206
00:09:19.485 --> 00:09:22.665
We also have the AI to give us a rationale, an explanation

207
00:09:22.685 --> 00:09:24.425
for why thinks it should pass or fail.

208
00:09:25.185 --> 00:09:26.845
So that's an exciting aspect of our study.

209
00:09:26.905 --> 00:09:28.645
We call this the narrative ai.

210
00:09:29.105 --> 00:09:32.245
Notice that when we ask Gent P to explain its decision,

211
00:09:32.755 --> 00:09:35.285
this is not how the model really thinks.

212
00:09:35.715 --> 00:09:39.725
This is a probabilistic explanation of what the model

213
00:09:40.295 --> 00:09:41.725
would say about this solution,

214
00:09:41.725 --> 00:09:43.805
meaning it's a potential explanation

215
00:09:43.825 --> 00:09:45.205
for why I think it should fail,

216
00:09:45.585 --> 00:09:48.805
but it does not describe how the weight in the model,

217
00:09:48.805 --> 00:09:50.925
the transformer, this AI brain is really working.

218
00:09:50.985 --> 00:09:52.485
So that's also something interesting

219
00:09:52.485 --> 00:09:55.085
that the AI can tell you why it thinks it failed,

220
00:09:55.385 --> 00:09:57.725
but this explanation does not reflect

221
00:09:57.725 --> 00:09:59.245
how the MO actually made the decision.

222
00:09:59.625 --> 00:10:01.045
So that's an interesting twist

223
00:10:01.745 --> 00:10:02.795
because we wanted to know

224
00:10:02.795 --> 00:10:05.035
how humans always interact with the text part.

225
00:10:05.305 --> 00:10:06.995
They never check what's happening in the weight.

226
00:10:06.995 --> 00:10:08.995
It's impossible for most of us.

227
00:10:10.055 --> 00:10:13.355
And now that we have all of this, we do a field experiments.

228
00:10:13.715 --> 00:10:15.915
LGPT is based on architecture,

229
00:10:16.035 --> 00:10:18.635
a neural network architecture called a transformer.

230
00:10:19.015 --> 00:10:22.095
So this is basically millions

231
00:10:22.235 --> 00:10:26.175
and billions of numbers that are all multiplied, added.

232
00:10:27.115 --> 00:10:30.015
Um, expon together differentiate,

233
00:10:30.015 --> 00:10:31.655
and they do this in multiple layers.

234
00:10:31.915 --> 00:10:35.725
So for instance, you give your sentence, that's the text

235
00:10:35.835 --> 00:10:37.485
that you, we call a prompt.

236
00:10:37.945 --> 00:10:39.365
It goes through those weights.

237
00:10:39.655 --> 00:10:40.685
Those weights are going

238
00:10:40.685 --> 00:10:43.845
to take every single word in your sentence, multiply it,

239
00:10:43.865 --> 00:10:45.685
add it all together in multiple different ways.

240
00:10:46.195 --> 00:10:49.245
It's going to pass it to a second layer

241
00:10:49.305 --> 00:10:50.925
of weights, those weights.

242
00:10:50.925 --> 00:10:52.965
So those numbers are going to do the same thing again,

243
00:10:53.605 --> 00:10:55.765
multiplying, adding all these first layer of numbers

244
00:10:56.185 --> 00:10:58.205
and then they're going to pass it to the next one

245
00:10:58.705 --> 00:10:59.965
and then to the next one.

246
00:11:00.025 --> 00:11:02.845
And multiple times and many times also in parallel,

247
00:11:03.795 --> 00:11:06.685
this ability to transform your data with tons

248
00:11:06.685 --> 00:11:08.325
of multiplications additions

249
00:11:08.385 --> 00:11:12.205
and other little functions is what makes the AI intelligent.

250
00:11:12.755 --> 00:11:14.405
It's basically able

251
00:11:14.505 --> 00:11:17.445
to have a very sophisticated pattern matching.

252
00:11:17.755 --> 00:11:20.645
It's able to learn statistics through all the training data

253
00:11:21.025 --> 00:11:22.485
to so many numbers that are able

254
00:11:22.485 --> 00:11:24.325
to add, multiply each other.

255
00:11:24.665 --> 00:11:28.005
So this is this AI brain and what we call the weights.

256
00:11:28.005 --> 00:11:30.085
Now that we have those decisions from the ai,

257
00:11:30.085 --> 00:11:31.925
we have recommendations and explanation.

258
00:11:32.385 --> 00:11:35.445
We want to see how humans are going to react to those,

259
00:11:35.505 --> 00:11:36.685
how are they going to use them.

260
00:11:37.185 --> 00:11:39.205
So we create our own platform.

261
00:11:39.465 --> 00:11:40.645
We created a little app

262
00:11:41.215 --> 00:11:45.845
where we will have people from multiple backgrounds

263
00:11:46.395 --> 00:11:49.445
that will evaluate actual real world solutions.

264
00:11:50.075 --> 00:11:53.645
Some of them will receive the help from the ai, some

265
00:11:53.645 --> 00:11:55.125
of them won't have any help.

266
00:11:55.955 --> 00:11:59.015
The people will receive help, some of them will have access

267
00:11:59.075 --> 00:12:03.215
to the AI recommendation, but not the explanation.

268
00:12:03.675 --> 00:12:06.015
And some will also have access to the explanation.

269
00:12:06.725 --> 00:12:10.945
And we wanted to measure how people are impacted by the fact

270
00:12:10.945 --> 00:12:14.585
that they have the ai, if they have the explanation on top.

271
00:12:15.585 --> 00:12:17.805
And then we did this with multiple people

272
00:12:17.955 --> 00:12:19.765
with four field experiments.

273
00:12:20.505 --> 00:12:24.525
One was with the actual screeners from MIT solve,

274
00:12:24.525 --> 00:12:25.765
meaning they do this all the time.

275
00:12:26.785 --> 00:12:29.805
One was with the community solvers, meaning people

276
00:12:29.865 --> 00:12:31.725
who are sometimes asked to be volunteers

277
00:12:31.785 --> 00:12:33.245
to help when they are overwhelmed.

278
00:12:34.305 --> 00:12:36.495
Third, my University of Washington students.

279
00:12:36.915 --> 00:12:38.855
So I was teaching generative AI

280
00:12:38.955 --> 00:12:41.495
and business data analytics to my undergrad students.

281
00:12:42.195 --> 00:12:44.495
And as an assignment, I asked them to do the study.

282
00:12:44.515 --> 00:12:47.095
So we also have undergrads who are not familiar

283
00:12:47.125 --> 00:12:50.055
with startup world at all in general,

284
00:12:50.955 --> 00:12:52.135
nor the whole process.

285
00:12:52.275 --> 00:12:55.455
So we are really coming to this world of evaluating.

286
00:12:55.805 --> 00:12:57.855
They have an AI and it's very interesting.

287
00:12:57.915 --> 00:13:00.055
Can we upskill people in evaluating with that?

288
00:13:00.775 --> 00:13:03.595
And then students from the Aspire Leaders program,

289
00:13:03.865 --> 00:13:04.955
they can be from high school

290
00:13:05.055 --> 00:13:07.195
or early undergrad from all over the world,

291
00:13:07.655 --> 00:13:10.875
and they dream to go study in the US in particular Harvard.

292
00:13:11.655 --> 00:13:14.555
And then they really want to be able to learn cool things.

293
00:13:14.655 --> 00:13:17.035
So we had those four field experiments.

294
00:13:17.415 --> 00:13:20.395
We collected amazing data, we preregistered the study.

295
00:13:20.735 --> 00:13:22.115
And now that we have the whole data,

296
00:13:23.005 --> 00:13:24.545
we actually analyze the data

297
00:13:24.685 --> 00:13:26.545
to derive some interesting conclusions.

298
00:13:27.305 --> 00:13:30.305
Whenever you have the ai, you tend to become harsher.

299
00:13:30.805 --> 00:13:34.465
And this is because our AI is the harshest of humans

300
00:13:34.845 --> 00:13:38.585
or human ai because we calibrate the AI to be harsh.

301
00:13:39.085 --> 00:13:41.545
So that's very interesting to know that you have

302
00:13:41.545 --> 00:13:44.825
to really think carefully about which AI you deploy.

303
00:13:45.405 --> 00:13:48.305
You want to deploy an AI that is overall as aligned

304
00:13:48.405 --> 00:13:50.785
as possible with your initial values.

305
00:13:51.085 --> 00:13:54.305
So we trained our AI to reflect

306
00:13:54.765 --> 00:13:58.145
how the professional solvers were screening

307
00:13:58.165 --> 00:13:59.625
out in the previous years.

308
00:14:00.165 --> 00:14:01.185
So we calibrated

309
00:14:01.725 --> 00:14:03.705
and we, we realized that naturally people

310
00:14:04.365 --> 00:14:06.865
are kinder than they should be in this process.

311
00:14:07.005 --> 00:14:08.745
We, they need to be harsher in this selection.

312
00:14:08.765 --> 00:14:12.385
So having the AI was able to realign people more

313
00:14:12.765 --> 00:14:14.225
to what's the goal of sprint?

314
00:14:14.325 --> 00:14:15.905
So that's first insight.

315
00:14:16.445 --> 00:14:18.305
And the more they ha if they have the explanation,

316
00:14:18.305 --> 00:14:19.785
it also make them even harsher.

317
00:14:20.125 --> 00:14:23.345
So they felt more convinced by having an explanation.

318
00:14:24.845 --> 00:14:28.945
Second insight, this can change depending of

319
00:14:29.485 --> 00:14:31.745
the type of criterion they have to evaluate.

320
00:14:31.765 --> 00:14:33.665
If you want to evaluate something objective

321
00:14:34.245 --> 00:14:36.385
or something subjective, the impact

322
00:14:36.385 --> 00:14:37.745
of the AI is not the same.

323
00:14:38.285 --> 00:14:39.385
We realize that if you want

324
00:14:39.385 --> 00:14:41.785
to evaluate a subjective criteria, for instance,

325
00:14:41.805 --> 00:14:45.725
do you think this solution is worth reading

326
00:14:45.945 --> 00:14:47.605
by an external reviewer?

327
00:14:47.785 --> 00:14:50.605
That's an actual criterion. It's super subjective.

328
00:14:50.705 --> 00:14:52.725
You imagine you've never screened ideas.

329
00:14:53.065 --> 00:14:54.565
How do you know that this idea is good enough

330
00:14:54.565 --> 00:14:56.285
that someone else outside

331
00:14:56.285 --> 00:14:58.405
of MIT solve should take the time to read it?

332
00:14:58.745 --> 00:15:02.405
That's really your human intuition and personal preference.

333
00:15:03.225 --> 00:15:05.965
And then whenever the AI enters this loop

334
00:15:06.545 --> 00:15:09.725
and then suggest you to maybe give it to

335
00:15:10.385 --> 00:15:13.285
an external reviewer, people will feel more convinced

336
00:15:13.835 --> 00:15:16.165
than if they just have the AI recommendation

337
00:15:17.025 --> 00:15:19.635
that feels intuitive in the sense that it's easier

338
00:15:19.775 --> 00:15:22.315
to influence you in something subjective if you don't

339
00:15:22.315 --> 00:15:24.915
necessarily have any benchmark yourself.

340
00:15:24.945 --> 00:15:28.115
Then we confirm that even the experts

341
00:15:29.205 --> 00:15:31.375
were influenced the same way as non-experts.

342
00:15:32.275 --> 00:15:34.615
And here by expert I mean domain experts.

343
00:15:35.035 --> 00:15:37.495
So people who have been doing this for maybe 10 years

344
00:15:38.615 --> 00:15:41.115
are equally influenced by the AI decisions

345
00:15:41.175 --> 00:15:42.275
as people who've done this

346
00:15:42.275 --> 00:15:44.115
for the very first time in the exercise.

347
00:15:44.115 --> 00:15:45.195
That's fascinating to see

348
00:15:45.195 --> 00:15:49.595
that your domain expertise in a way can be overridden

349
00:15:49.655 --> 00:15:51.475
by the AI advice

350
00:15:53.625 --> 00:15:56.325
and, all, we have a lot of other exciting things.

351
00:15:56.405 --> 00:15:57.845
I want to give this other one that we,

352
00:15:57.925 --> 00:16:01.685
I really like the fact that the people who were most able

353
00:16:02.265 --> 00:16:03.885
to override the AI decision

354
00:16:03.985 --> 00:16:07.005
or to agree with the AI when it was relevant were the people

355
00:16:07.505 --> 00:16:10.445
who interacted the most with the solution

356
00:16:10.445 --> 00:16:14.685
and the AI decision, meaning that the people who really try

357
00:16:14.685 --> 00:16:17.085
to understand the tool try to leverage

358
00:16:17.085 --> 00:16:19.685
and understand the mechanism of the AI recommendation.

359
00:16:20.355 --> 00:16:21.685
Were the most critical

360
00:16:22.265 --> 00:16:26.525
and the most able to leverage the full potential of the ai.

361
00:16:26.525 --> 00:16:30.565
Suggesting that there is strategy into teaching people how

362
00:16:30.565 --> 00:16:33.485
to use the tool, how to design a tool

363
00:16:33.485 --> 00:16:35.205
that people can leverage.

364
00:16:35.785 --> 00:16:38.245
And it's really good insights, managerial insights

365
00:16:38.245 --> 00:16:40.085
for whoever is wanting to build a pipeline

366
00:16:40.085 --> 00:16:42.485
to help having the AI in the loop.

367
00:16:42.595 --> 00:16:46.245
What we see is the emergence of a division of labor

368
00:16:46.625 --> 00:16:48.605
around the fact we can evaluate ideas.

369
00:16:48.745 --> 00:16:51.925
In particular, it looks like it's easier to automate

370
00:16:52.465 --> 00:16:55.125
an objective decision than a subjective one.

371
00:16:55.355 --> 00:16:58.565
Meaning that if we properly calibrate an AI model,

372
00:16:59.145 --> 00:17:00.845
we can sort of align it

373
00:17:01.365 --> 00:17:04.925
whenever it's objective, meaning that the AI has clear ways

374
00:17:05.025 --> 00:17:07.965
of figuring out if this idea is passing

375
00:17:08.115 --> 00:17:09.325
this objective criteria.

376
00:17:09.425 --> 00:17:13.125
An example is, is your idea already in a prototype stage

377
00:17:13.625 --> 00:17:16.045
or is your idea powered by technology?

378
00:17:16.225 --> 00:17:18.165
So MIT so only wants to fund ideas

379
00:17:18.165 --> 00:17:19.405
that are at least the prototype

380
00:17:19.945 --> 00:17:23.005
and use technology as the core of our product.

381
00:17:23.075 --> 00:17:27.495
It's MIT. This is something you can fact check

382
00:17:27.835 --> 00:17:29.535
and you can have, you can back it

383
00:17:29.535 --> 00:17:32.915
with specific information from the solution there.

384
00:17:33.095 --> 00:17:35.675
The AI can automate your process potentially.

385
00:17:35.975 --> 00:17:37.715
And this comes back to an earlier question

386
00:17:37.715 --> 00:17:39.995
that we can focus on in the creativity

387
00:17:40.695 --> 00:17:44.675
and we delegate the hard work in a way to the ai.

388
00:17:44.695 --> 00:17:46.435
So that's one insight that we confirmed here.

389
00:17:47.835 --> 00:17:48.855
Second, insight,

390
00:17:49.685 --> 00:17:51.565
Whenever it involves the subjectivity,

391
00:17:52.535 --> 00:17:53.955
we don't want to remove the humans.

392
00:17:54.255 --> 00:17:57.835
We want to have them to be augmented, meaning I want

393
00:17:57.835 --> 00:17:59.755
to provide some assistance into

394
00:17:59.895 --> 00:18:01.955
how they can review this solution faster

395
00:18:02.905 --> 00:18:06.925
and in a more, in a manner that will feel fair to everyone.

396
00:18:07.125 --> 00:18:10.365
Because actually it's not that fair to have

397
00:18:11.125 --> 00:18:12.845
multiple screeners that can be randomized

398
00:18:13.025 --> 00:18:14.125
by the fact that you don't.

399
00:18:14.275 --> 00:18:16.645
They only have two minutes and a half per evaluation.

400
00:18:16.645 --> 00:18:18.845
That's so little. So they really have

401
00:18:18.845 --> 00:18:20.325
to leverage their expertise super fast.

402
00:18:20.745 --> 00:18:22.685
The AI will read everything.

403
00:18:23.145 --> 00:18:26.245
It won't take any bias of not reading that

404
00:18:26.285 --> 00:18:28.525
or think, okay, this is not worth reading.

405
00:18:28.585 --> 00:18:30.005
It would just read all of it.

406
00:18:30.465 --> 00:18:32.645
So it brings more fairness somehow if

407
00:18:32.645 --> 00:18:33.725
you properly calibrate.

408
00:18:33.785 --> 00:18:37.885
So this is also the insight that various advantages using AI

409
00:18:38.025 --> 00:18:41.005
to help you process huge amount of information.

410
00:18:41.745 --> 00:18:44.525
And finally, you need to tell people how

411
00:18:44.525 --> 00:18:47.365
to properly use a tool, how to calibrate, how

412
00:18:47.365 --> 00:18:50.965
to create a strong synergy between the human and the ai.

413
00:18:51.425 --> 00:18:53.325
And it's not random. It's not enough

414
00:18:53.325 --> 00:18:55.165
to develop a great AI tool by itself

415
00:18:55.315 --> 00:18:57.205
because as I mentioned earlier,

416
00:18:57.545 --> 00:19:00.485
you can become worse than if you did not have any AI

417
00:19:00.705 --> 00:19:02.125
or worse than the AI alone.

418
00:19:02.125 --> 00:19:05.165
So you need work to figure out a proper design

419
00:19:05.465 --> 00:19:06.845
of a human AI collaboration.

Exploring AI's Role in Problem Solving and Innovation

00:00:00 - I'm researching the future of work. I want to know how technologies such as artificial intelligence will impact the way we work. My goal is to accelerate problem solving. I want to help more people to achieve more things. And in particular, I'm focusing these days on multiple stages of how can we solve big problems like the sustainable development goals by the United Nations. One part of solving problems is to find great ideas to solving challenging problems,

00:00:34 - You need to go from a spark of an idea to an actual prototype, realiz realization, pro product. I want to build AI tools and tech technological tools to help people in every stage. The research that we're going to talk about today is around the early stages of innovation. So a previous study that we've done with my colleagues at Harvard is around how can we leverage generative AI

00:01:04 - And the whole process of human AI collaboration for finding ideas to challenging problems. The project we are going to talk about today is extremely important for any company, organization, students, learners who are facing many, many choices, and they wish to have the help of a tool like AI to help them select the best ideas, the ideas that they should push more, interest in. They're collaborating with MIT Solve.

Leveraging AI for Global Problem Solving

00:01:35 - They give about 70, they've been giving $70 million in funding to so many, early stage projects all over the world to help, them solve things around financial inclusion, global health, education for all. And they receive thousands of applications to their crowdsourcing challenges. So their goal is to get solutions from all over the world, but then they have thousands of people to evaluate,

00:02:09 - Their goal is to go from those thousands of solutions to a subset of maybe a hundred or 200 that they shortlisted, and then they want to find which ones should be the finalists and which one should be the winners. It's a super time consuming process and it's very overwhelming to be able to ski through so much content. So the whole point of this paper is to build a human AI collaboration to help us

00:02:37 - The whole point is the human AI collaboration. We don't want to replace humans. We want to augment our decision making. That meaning have the people who will listen to these have used chat pt, maybe they created images, for instance with Mid Journey or Dali. Basically, the computer science literature focuses on deploying better technology, but they don't always necessarily study how those technologies will impact the future of work or businesses. This is where there is another literature entering, which is

Optimizing Human-AI Collaboration: Benefits and Challenges

00:03:11 - How are they using it? What's the optimal way, the optimal division of labor with the ai? So there is a literature showing that collaborating with AI can make you obviously better. It can help upskill people. It can help you do tasks that were deemed infeasible before. However, in certain situations, it can also make you worse because you are sort of outside of the realm of what AI can do. And then you should have relied on your own expertise versus

00:03:41 - So there is a literature on that. There is also a literature on studying in which scenarios having a human and an AI collaborating together will outperform the AI alone and the human alone. And it's absolutely not guaranteed to have a situation like this. It's really possible that human AI outperforms the human only, but the AI alone would fare better or the human would fare better than the human ai. So this is where there is the literature.

00:04:11 - what happens when we do evaluations, which is something that has not inve been investigated much yet. I am a huge advocate of human creativity. I love being able to figure out how to solve problems, but I'm also a huge advocate of leveraging technology to make us even more creative. And then what I witnessed is my students, like I teach generative AI to hundreds of students every year, to companies, to organizations.

Rekindling Creativity Through AI

00:04:44 - because we don't think ourselves as artists because we don't think of extra use cases. So one of the key things in my teaching is I want people to feel like artists again, because I believe that using generative AI feels like playing with a tool like a brush. And I want people to feel that they can innovate through their way of using ai, and they can find new cases that nobody has investigated before.

00:05:13 - for even more creativity, and I believe that this is how you can get some exponential growth solving because we collaborate, but we need the human ingenuity, the human craft thinking to get there. I believe that we won't be able to remove the human in the loop. And I, I think there are two things that can happen. AI in the loop, human in the loop and variations, but basically some AI systems, they try to remove

00:05:43 - So if you look at the latest model from church PTs or one one, this model is supposedly thinking. So OpenAI says it's thinking, it's actually not thinking. It's more like processing information. And then you don't see exactly what it thinks. You see like a chain of thought. So you see like, what's the model doing right now? You don't see the details and it just gives you an output that is probably much better. if it's around math and science than many models before, it's very impress. I've very, I've been very, I impressive with some use cases.

Balancing Human and AI Collaboration in Innovation

00:06:15 - of remove the human in the middle and you can iterate with the final content, but not really with what's happening in the middle. I am sure that use cases like this will happen. So I agree that in some scenarios we are going to remove the human because it's really in the range of what AI can do very well. We will just need to be able to check. So that's important. I think that humans can have the control on what's happening, but then the humans will work with the final part. Yet I think we, I'm not seeing the technology

00:06:47 - to focus on creativity. We'll have to focus on interacting as well, or at least there will be a decent amount of time, maybe at least 3, 4, 5 years if if not more. And I believe more where we have to keep collaborating all along the process. And I believe that this is one of the points that we are making in the paper, that we have domain expertise. So that's traditional knowledge of a given field. For instance, physics, math, computer science, innovation. But also we see the emergence of a new type of expertise,

00:07:23 - with the AI to get the AI to do what you want. And this is not only creativity, it's also a skill. The ability to talk to the AI to shift, to nudge the ai, to do what you really wanted to do. Our first goal was to actually collaborate with a real world platform. We don't want to do work at theoretical. And even though we would do field experiments, we would get insights. We really wanted to use real world data and help actual companies or organizations.

Evaluating Ideas at Scale with AI Collaboration

00:07:53 - They were amazing because they gave us an amount of data that is beautiful and this data is so precious. It's people who've been working to make the world a better place, and we have thousands of such ideas. So that was the first principle, a great collaboration. Second, we wanted to understand how they actually evaluate ideas at scale. And so what they do is that they hire, externally, they, they have volunteers, committee volunteers

00:08:24 - and they also have internal judges and staff to go through thousands of solutions every time is a new challenge. And then they have a clean process of doing this. They have a hierarchy of criteria that every idea has to pass, and then it goes through multiple screeners, et cetera. We wanted to replicate this entire pipeline with large language models. In particular, we use GPT-4, some sophisticated prompt engineering to be able to tell the model how to evaluate an idea. And thanks to this we collected for every idea.

00:08:58 - to 15 pages about what's your solution about who's working on it, and are you at a prototype stage, extract the, the technology behind, et cetera. We have an AI that can read all that and tell us if it thinks the solution will pass or fail every criterion. So now we have AI recommendations. We also have the AI to give us a rationale, an explanation for why thinks it should pass or fail. So that's an exciting aspect of our study.

Understanding AI's Decision-Making Process

00:09:29 - Notice that when we ask Gent P to explain its decision, this is not how the model really thinks. This is a probabilistic explanation of what the model would say about this solution, meaning it's a potential explanation for why I think it should fail, but it does not describe how the weight in the model, the transformer, this AI brain is really working. So that's also something interesting that the AI can tell you why it thinks it failed, but this explanation does not reflect

00:09:59 - So that's an interesting twist because we wanted to know how humans always interact with the text part. They never check what's happening in the weight. It's impossible for most of us. And now that we have all of this, we do a field experiments. LGPT is based on architecture, a neural network architecture called a transformer. So this is basically millions and billions of numbers that are all multiplied, added. expon together differentiate,

00:10:31 - So for instance, you give your sentence, that's the text that you, we call a prompt. It goes through those weights. Those weights are going to take every single word in your sentence, multiply it, add it all together in multiple different ways. It's going to pass it to a second layer of weights, those weights. So those numbers are going to do the same thing again, multiplying, adding all these first layer of numbers and then they're going to pass it to the next one and then to the next one.

Exploring AI Decision-Making and Human Interaction

00:11:03 - this ability to transform your data with tons of multiplications additions and other little functions is what makes the AI intelligent. It's basically able to have a very sophisticated pattern matching. It's able to learn statistics through all the training data to so many numbers that are able to add, multiply each other. So this is this AI brain and what we call the weights. Now that we have those decisions from the ai,

00:11:32 - We want to see how humans are going to react to those, how are they going to use them. So we create our own platform. We created a little app where we will have people from multiple backgrounds that will evaluate actual real world solutions. Some of them will receive the help from the ai, some of them won't have any help. The people will receive help, some of them will have access to the AI recommendation, but not the explanation.

00:12:06 - And we wanted to measure how people are impacted by the fact that they have the ai, if they have the explanation on top. And then we did this with multiple people with four field experiments. One was with the actual screeners from MIT solve, meaning they do this all the time. One was with the community solvers, meaning people who are sometimes asked to be volunteers to help when they are overwhelmed.

Enhancing Evaluation Skills with AI

00:12:36 - So I was teaching generative AI and business data analytics to my undergrad students. And as an assignment, I asked them to do the study. So we also have undergrads who are not familiar with startup world at all in general, nor the whole process. So we are really coming to this world of evaluating. They have an AI and it's very interesting. Can we upskill people in evaluating with that? And then students from the Aspire Leaders program, they can be from high school

00:13:07 - and they dream to go study in the US in particular Harvard. And then they really want to be able to learn cool things. So we had those four field experiments. We collected amazing data, we preregistered the study. And now that we have the whole data, we actually analyze the data to derive some interesting conclusions. Whenever you have the ai, you tend to become harsher. And this is because our AI is the harshest of humans or human ai because we calibrate the AI to be harsh.

00:13:41 - to really think carefully about which AI you deploy. You want to deploy an AI that is overall as aligned as possible with your initial values. So we trained our AI to reflect how the professional solvers were screening out in the previous years. So we calibrated and we, we realized that naturally people are kinder than they should be in this process. We, they need to be harsher in this selection. So having the AI was able to realign people more

AI's Impact on Subjective Evaluation

00:14:14 - So that's first insight. And the more they ha if they have the explanation, it also make them even harsher. So they felt more convinced by having an explanation. Second insight, this can change depending of the type of criterion they have to evaluate. If you want to evaluate something objective or something subjective, the impact of the AI is not the same. We realize that if you want to evaluate a subjective criteria, for instance, do you think this solution is worth reading

00:14:47 - That's an actual criterion. It's super subjective. You imagine you've never screened ideas. How do you know that this idea is good enough that someone else outside of MIT solve should take the time to read it? That's really your human intuition and personal preference. And then whenever the AI enters this loop and then suggest you to maybe give it to an external reviewer, people will feel more convinced than if they just have the AI recommendation

00:15:19 - to influence you in something subjective if you don't necessarily have any benchmark yourself. Then we confirm that even the experts were influenced the same way as non-experts. And here by expert I mean domain experts. So people who have been doing this for maybe 10 years are equally influenced by the AI decisions as people who've done this for the very first time in the exercise. That's fascinating to see that your domain expertise in a way can be overridden

Maximizing AI Effectiveness Through User Engagement

00:15:53 - and, all, we have a lot of other exciting things. I want to give this other one that we, I really like the fact that the people who were most able to override the AI decision or to agree with the AI when it was relevant were the people who interacted the most with the solution and the AI decision, meaning that the people who really try to understand the tool try to leverage and understand the mechanism of the AI recommendation.

00:16:22 - and the most able to leverage the full potential of the ai. Suggesting that there is strategy into teaching people how to use the tool, how to design a tool that people can leverage. And it's really good insights, managerial insights for whoever is wanting to build a pipeline to help having the AI in the loop. What we see is the emergence of a division of labor around the fact we can evaluate ideas. In particular, it looks like it's easier to automate

00:16:55 - Meaning that if we properly calibrate an AI model, we can sort of align it whenever it's objective, meaning that the AI has clear ways of figuring out if this idea is passing this objective criteria. An example is, is your idea already in a prototype stage or is your idea powered by technology? So MIT so only wants to fund ideas that are at least the prototype and use technology as the core of our product.

Balancing AI Automation and Human Insight

00:17:27 - and you can have, you can back it with specific information from the solution there. The AI can automate your process potentially. And this comes back to an earlier question that we can focus on in the creativity and we delegate the hard work in a way to the ai. So that's one insight that we confirmed here. Second, insight, Whenever it involves the subjectivity, we don't want to remove the humans.

00:17:57 - to provide some assistance into how they can review this solution faster and in a more, in a manner that will feel fair to everyone. Because actually it's not that fair to have multiple screeners that can be randomized by the fact that you don't. They only have two minutes and a half per evaluation. That's so little. So they really have to leverage their expertise super fast. The AI will read everything. It won't take any bias of not reading that

00:18:28 - It would just read all of it. So it brings more fairness somehow if you properly calibrate. So this is also the insight that various advantages using AI to help you process huge amount of information. And finally, you need to tell people how to properly use a tool, how to calibrate, how to create a strong synergy between the human and the ai. And it's not random. It's not enough to develop a great AI tool by itself because as I mentioned earlier,