1
00:00:00,070 --> 00:00:02,500
The following content is
provided under a Creative

2
00:00:02,500 --> 00:00:04,019
Commons license.

3
00:00:04,019 --> 00:00:06,360
Your support will help
MIT OpenCourseWare

4
00:00:06,360 --> 00:00:10,730
continue to offer high quality
educational resources for free.

5
00:00:10,730 --> 00:00:13,330
To make a donation or
view additional materials

6
00:00:13,330 --> 00:00:17,236
from hundreds of MIT courses,
visit MIT OpenCourseWare

7
00:00:17,236 --> 00:00:17,861
at ocw.mit.edu.

8
00:00:23,157 --> 00:00:23,740
PROFESSOR: OK.

9
00:00:23,740 --> 00:00:24,600
Welcome to class.

10
00:00:27,890 --> 00:00:31,100
The theme of today's lecture
is all about testing.

11
00:00:31,100 --> 00:00:35,010
So I'm going to give the
intro to testing lecture

12
00:00:35,010 --> 00:00:38,270
where I talk about testing
and how, what it is,

13
00:00:38,270 --> 00:00:39,026
and why we do it.

14
00:00:39,026 --> 00:00:40,400
Give you some
basic testing tips,

15
00:00:40,400 --> 00:00:42,733
and let you know what we
expect you to be doing in terms

16
00:00:42,733 --> 00:00:44,710
of testing for your projects.

17
00:00:44,710 --> 00:00:48,016
And I'll spend a little
more time on focus testing,

18
00:00:48,016 --> 00:00:49,890
because that's the part
people are frequently

19
00:00:49,890 --> 00:00:52,660
least familiar with.

20
00:00:52,660 --> 00:00:54,380
We have Genevieve
Conley from Riot,

21
00:00:54,380 --> 00:00:58,150
and she's going to give a
lecture related to testing.

22
00:00:58,150 --> 00:01:01,160
But more about what Riot's
doing and stuff like that.

23
00:01:01,160 --> 00:01:03,270
So hopefully that will be fun.

24
00:01:03,270 --> 00:01:04,590
We'll take a break.

25
00:01:04,590 --> 00:01:06,847
And then since we just
told you all about testing

26
00:01:06,847 --> 00:01:09,430
and how important it is, we will
go ahead and run a focus test

27
00:01:09,430 --> 00:01:11,634
in class.

28
00:01:11,634 --> 00:01:13,550
We'll give you some time
to get things set up,

29
00:01:13,550 --> 00:01:15,770
probably about 10 minutes
to plan your test.

30
00:01:15,770 --> 00:01:18,920
Then we'll set things up
and we will run testing.

31
00:01:18,920 --> 00:01:22,020
We're going to ask that everyone
on your team test at least

32
00:01:22,020 --> 00:01:27,010
two games and that
everyone on your team

33
00:01:27,010 --> 00:01:29,770
observe at least one
person playing a game.

34
00:01:29,770 --> 00:01:33,880
How exactly you're going to
make all of that happen in flow

35
00:01:33,880 --> 00:01:36,636
is your problem.

36
00:01:36,636 --> 00:01:37,760
And then we will come down.

37
00:01:37,760 --> 00:01:40,640
We'll give you some time to let
your team compare observations

38
00:01:40,640 --> 00:01:42,440
and talk about
results, and then we'll

39
00:01:42,440 --> 00:01:44,670
ask you to come down for
a standard in class, two

40
00:01:44,670 --> 00:01:48,640
minute no visuals presentation.

41
00:01:48,640 --> 00:01:54,250
Monday, project two is due
along with your design change

42
00:01:54,250 --> 00:02:00,920
log and focus test reports
and your individual write-ups,

43
00:02:00,920 --> 00:02:02,560
and the project
two presentation.

44
00:02:02,560 --> 00:02:04,030
This is a five
minute presentation.

45
00:02:04,030 --> 00:02:05,904
There are presentation
guidelines on Stellar.

46
00:02:05,904 --> 00:02:07,940
You can download
and look at them.

47
00:02:07,940 --> 00:02:09,470
Once again, we're
primarily looking

48
00:02:09,470 --> 00:02:11,105
for postmortem and reflection.

49
00:02:11,105 --> 00:02:12,686
We want to hear
about your process.

50
00:02:12,686 --> 00:02:14,060
We want to know
what you learned.

51
00:02:14,060 --> 00:02:15,850
We want to know what you did
right, what you did wrong,

52
00:02:15,850 --> 00:02:17,650
and what you're going
to do in the future.

53
00:02:17,650 --> 00:02:19,460
And we want it in five minutes.

54
00:02:22,510 --> 00:02:28,220
Grades for project one are
up, along with comments

55
00:02:28,220 --> 00:02:30,820
about your individual
postmortems

56
00:02:30,820 --> 00:02:34,450
and some comments about
the overall project stuff.

57
00:02:34,450 --> 00:02:36,850
Overall, we were really
impressed with the quality

58
00:02:36,850 --> 00:02:38,290
of project one.

59
00:02:38,290 --> 00:02:40,700
So keep it up.

60
00:02:40,700 --> 00:02:42,620
But do you stop and
read the comments

61
00:02:42,620 --> 00:02:45,200
so that you guys
can make sure to do

62
00:02:45,200 --> 00:02:49,540
good jobs on your individual
postmortems going forward.

63
00:02:49,540 --> 00:02:51,550
Anything to add?

64
00:02:51,550 --> 00:02:54,100
All right.

65
00:02:54,100 --> 00:02:58,180
So people say testing.

66
00:02:58,180 --> 00:02:59,630
People say quality assurance.

67
00:02:59,630 --> 00:03:01,161
What are they?

68
00:03:01,161 --> 00:03:03,285
Quality assurance is actually
just a whole process.

69
00:03:03,285 --> 00:03:05,570
It involves a whole lot of
different types of testing.

70
00:03:05,570 --> 00:03:11,140
But in general, it's
the systematic process

71
00:03:11,140 --> 00:03:13,360
checking to see whether
a product of service

72
00:03:13,360 --> 00:03:17,290
being developed is meeting
its specified requirements.

73
00:03:17,290 --> 00:03:19,190
The most important
part of that definition

74
00:03:19,190 --> 00:03:21,500
really is systemic,
which means you're doing

75
00:03:21,500 --> 00:03:23,690
regular, repeatable testing.

76
00:03:23,690 --> 00:03:27,710
Looking at making sure that
all the portions of your game

77
00:03:27,710 --> 00:03:28,920
or project are working.

78
00:03:28,920 --> 00:03:30,660
And requirements,
making sure you're

79
00:03:30,660 --> 00:03:34,700
meeting that it
doesn't just "work"

80
00:03:34,700 --> 00:03:36,310
as far as you think it does.

81
00:03:36,310 --> 00:03:38,370
It works according the
requirements of the game,

82
00:03:38,370 --> 00:03:40,760
of the project, of
whoever you're shipping

83
00:03:40,760 --> 00:03:42,010
to and et cetera and so forth.

84
00:03:42,010 --> 00:03:44,887
So you need to know what
your standards are to be

85
00:03:44,887 --> 00:03:46,095
doing good quality assurance.

86
00:03:50,830 --> 00:03:53,940
Without testing, your
project will fail,

87
00:03:53,940 --> 00:03:57,350
and will take more time, and
will take a lot more effort.

88
00:03:57,350 --> 00:03:59,740
You can make games
without programmers.

89
00:03:59,740 --> 00:04:02,702
There's a lot of graphical user
interface engines out there.

90
00:04:02,702 --> 00:04:04,410
Some of them are
getting older and older,

91
00:04:04,410 --> 00:04:06,201
but they're out there
and you can use them.

92
00:04:06,201 --> 00:04:07,190
GameMaker, for example.

93
00:04:07,190 --> 00:04:08,350
You can make games
without artists.

94
00:04:08,350 --> 00:04:09,891
I bet several of
you are figuring out

95
00:04:09,891 --> 00:04:11,330
how to do that right now.

96
00:04:11,330 --> 00:04:13,680
But you can also
make games completely

97
00:04:13,680 --> 00:04:16,890
without assets using text
engines like Inform 7.

98
00:04:19,184 --> 00:04:21,600
You don't really need designated
producer or distinct game

99
00:04:21,600 --> 00:04:23,290
designer on a small team.

100
00:04:23,290 --> 00:04:25,500
One person can
really do it alone.

101
00:04:25,500 --> 00:04:28,790
But you're never going to make
a good game without testing,

102
00:04:28,790 --> 00:04:31,690
and the more
systemic and planned

103
00:04:31,690 --> 00:04:34,890
your testing is, the
more results you'll

104
00:04:34,890 --> 00:04:36,210
get from less effort.

105
00:04:39,980 --> 00:04:44,020
So who gets to do
all this testing?

106
00:04:44,020 --> 00:04:46,960
That's really everyone
involved in the project.

107
00:04:46,960 --> 00:04:49,330
Do not assume that
because you are

108
00:04:49,330 --> 00:04:51,100
the key programmer
or the key designer

109
00:04:51,100 --> 00:04:54,510
or the key whatever that someone
else will check your code

110
00:04:54,510 --> 00:04:58,290
and make sure it's working
after you've integrated it.

111
00:04:58,290 --> 00:05:00,170
You really want
everyone on the team

112
00:05:00,170 --> 00:05:01,940
to consider it to be
their responsibility

113
00:05:01,940 --> 00:05:04,320
to keep the game working
as well as possible

114
00:05:04,320 --> 00:05:07,120
all the way through, and
also to take the time

115
00:05:07,120 --> 00:05:09,180
to go in and check
other people's stuff

116
00:05:09,180 --> 00:05:12,080
to make sure that it went in.

117
00:05:12,080 --> 00:05:16,090
Remember that this is one of
the cornerstones of using Scrum

118
00:05:16,090 --> 00:05:18,990
as well, where before a
task is actually done,

119
00:05:18,990 --> 00:05:20,800
someone else has gone
in and verified it.

120
00:05:20,800 --> 00:05:23,300
Well, someone else is going to
be someone on the team, which

121
00:05:23,300 --> 00:05:26,620
means you're all going to have
allot some time to doing it,

122
00:05:26,620 --> 00:05:28,940
and you're going to have
to think about that.

123
00:05:28,940 --> 00:05:32,870
Speaking of which, anybody have
testing as one of your tasks

124
00:05:32,870 --> 00:05:34,163
on your sprint task list?

125
00:05:37,950 --> 00:05:39,680
Anybody?

126
00:05:39,680 --> 00:05:41,960
Woo, you win.

127
00:05:41,960 --> 00:05:43,340
It's going to take some time.

128
00:05:43,340 --> 00:05:45,800
And it's going to take some
time that probably otherwise

129
00:05:45,800 --> 00:05:48,210
could be used for making code.

130
00:05:48,210 --> 00:05:49,720
So go ahead and
remember that now

131
00:05:49,720 --> 00:05:51,219
and think about
that as you're going

132
00:05:51,219 --> 00:05:53,560
into your last week
of work, that you need

133
00:05:53,560 --> 00:05:57,070
to allot some time for testing.

134
00:05:57,070 --> 00:05:58,566
So types of testing.

135
00:05:58,566 --> 00:05:59,940
What is the
terminology I'm going

136
00:05:59,940 --> 00:06:04,550
to use so that we all understand
what I'm saying when I say it?

137
00:06:04,550 --> 00:06:08,230
I divide testing up
into three major areas,

138
00:06:08,230 --> 00:06:14,240
and there isn't a lot of formal
methodology on this, especially

139
00:06:14,240 --> 00:06:15,930
not in the game industry.

140
00:06:15,930 --> 00:06:19,170
So I tend to talk about
technical testing, play

141
00:06:19,170 --> 00:06:21,760
testing, and user testing.

142
00:06:21,760 --> 00:06:27,740
When I say technical testing, I
mean getting your game to run.

143
00:06:27,740 --> 00:06:29,910
Specifically, I often
mean getting your game

144
00:06:29,910 --> 00:06:33,070
to run exactly as how
you intended it to run.

145
00:06:35,760 --> 00:06:39,940
So if the designer claims that
the peach mobile should go up

146
00:06:39,940 --> 00:06:42,910
to 30 miles per hour and fire
eight peaches per second,

147
00:06:42,910 --> 00:06:44,630
if you go into the
game and you discover

148
00:06:44,630 --> 00:06:46,500
that it goes 70 miles
per hour and fires

149
00:06:46,500 --> 00:06:48,820
two peaches per
second, that's not

150
00:06:48,820 --> 00:06:52,140
working even if it makes game
awfully fun with it like that.

151
00:06:55,550 --> 00:06:58,047
When you start thinking
about whether it's fun or not

152
00:06:58,047 --> 00:07:00,380
as a developer, then you're
moving into the area of play

153
00:07:00,380 --> 00:07:02,980
testing which is
where you, the team,

154
00:07:02,980 --> 00:07:05,200
are playing your game
trying to figure out,

155
00:07:05,200 --> 00:07:08,622
hey, all those requirements and
constraints and plans we made,

156
00:07:08,622 --> 00:07:09,330
are they working?

157
00:07:09,330 --> 00:07:12,886
And are they working well, and
is a player going to enjoy it?

158
00:07:12,886 --> 00:07:14,760
You're play testing your
game as a developer.

159
00:07:14,760 --> 00:07:16,343
You're checking to
make sure it works.

160
00:07:16,343 --> 00:07:17,140
In that case, yes.

161
00:07:17,140 --> 00:07:18,931
The fact that the peach
mobile is currently

162
00:07:18,931 --> 00:07:23,250
going 30 miles per hour and it's
so slow that no player is ever

163
00:07:23,250 --> 00:07:25,100
going to want to use it.

164
00:07:25,100 --> 00:07:25,850
That's a bug.

165
00:07:25,850 --> 00:07:27,230
That's a really big bug.

166
00:07:27,230 --> 00:07:29,509
But it's a play testing
bug if your specs say

167
00:07:29,509 --> 00:07:31,050
it should be going
30 miles per hour,

168
00:07:31,050 --> 00:07:32,370
not a technical testing bug.

169
00:07:32,370 --> 00:07:34,940
And it is good to know the
differences so that you

170
00:07:34,940 --> 00:07:36,650
can know what
you're talking about

171
00:07:36,650 --> 00:07:38,640
and why you're talking about it.

172
00:07:38,640 --> 00:07:41,840
Finally, there's
user testing, which

173
00:07:41,840 --> 00:07:44,800
is getting someone who is not
on your team to play your game.

174
00:07:47,104 --> 00:07:49,020
You, the developer, while
you do play testing,

175
00:07:49,020 --> 00:07:50,500
can pretend to be
an actual player.

176
00:07:50,500 --> 00:07:52,110
But you are not
an actual player.

177
00:07:52,110 --> 00:07:54,400
You will never be an actual
player for your game.

178
00:07:54,400 --> 00:07:55,790
You know it too well.

179
00:07:55,790 --> 00:07:57,380
You are too close to it.

180
00:07:57,380 --> 00:07:59,270
You have spent too
many hours playing it.

181
00:08:02,120 --> 00:08:04,860
So you really need to
get someone who has never

182
00:08:04,860 --> 00:08:07,360
seen your game before, ideally,
to come in and sit down

183
00:08:07,360 --> 00:08:09,456
in front of your
game and play it.

184
00:08:09,456 --> 00:08:11,830
We're going to make pretend
when we run our focus testing

185
00:08:11,830 --> 00:08:16,360
workshop that your classmates
here are actually good testers.

186
00:08:16,360 --> 00:08:18,270
Tell you a secret, they're not.

187
00:08:18,270 --> 00:08:20,917
They've seen your game because
they've play tested it before.

188
00:08:20,917 --> 00:08:22,250
They've heard you talk about it.

189
00:08:22,250 --> 00:08:23,380
They heard you pitch it.

190
00:08:23,380 --> 00:08:25,860
They might even have worked
on it if they switched teams.

191
00:08:25,860 --> 00:08:27,740
So they're not actually
your ideal testers,

192
00:08:27,740 --> 00:08:29,350
but they are very
convenient testers.

193
00:08:29,350 --> 00:08:31,010
So we're going to
use them today.

194
00:08:31,010 --> 00:08:32,570
And for your next
projects, we're

195
00:08:32,570 --> 00:08:34,240
going to encourage
you-- and, in fact,

196
00:08:34,240 --> 00:08:36,510
require you-- to go out
into the cold, cold world

197
00:08:36,510 --> 00:08:39,850
and find actual testers
who haven't seen your game.

198
00:08:39,850 --> 00:08:40,799
People in your dorms.

199
00:08:40,799 --> 00:08:42,007
People in your other classes.

200
00:08:42,007 --> 00:08:44,860
Friends, people at
the coffee shop.

201
00:08:44,860 --> 00:08:47,030
Whoever you think will do
the best job of giving you

202
00:08:47,030 --> 00:08:48,071
the information you need.

203
00:08:51,640 --> 00:08:53,796
User testing is another
one of those terminologies

204
00:08:53,796 --> 00:08:55,460
that means a whole lot.

205
00:08:55,460 --> 00:08:57,930
So for my convenience,
I tend to divide it

206
00:08:57,930 --> 00:09:00,680
up into two kinds of things,
focus testing and "user

207
00:09:00,680 --> 00:09:02,000
testing."

208
00:09:02,000 --> 00:09:03,750
Where "focus testing"
is specifically

209
00:09:03,750 --> 00:09:05,958
when you're sitting down
and you're playing the game,

210
00:09:05,958 --> 00:09:07,120
looking at how it plays.

211
00:09:07,120 --> 00:09:07,882
Is it fun?

212
00:09:07,882 --> 00:09:08,840
Are people enjoying it?

213
00:09:08,840 --> 00:09:09,881
Is it challenging enough?

214
00:09:09,881 --> 00:09:12,320
Are the mechanics interesting?

215
00:09:12,320 --> 00:09:15,410
Is your game a good game, and
getting that information back

216
00:09:15,410 --> 00:09:16,700
from your players.

217
00:09:16,700 --> 00:09:19,940
When user testing,
I tend to use more

218
00:09:19,940 --> 00:09:22,640
for looking at the
usability of your game.

219
00:09:22,640 --> 00:09:24,690
You may have a really,
really fun game,

220
00:09:24,690 --> 00:09:26,650
but if it takes your
players 12 minutes

221
00:09:26,650 --> 00:09:29,450
to get off your starting page
because they can't figure out

222
00:09:29,450 --> 00:09:31,670
what they're doing
or how to do it,

223
00:09:31,670 --> 00:09:33,990
that's a pretty serious
usability issue.

224
00:09:33,990 --> 00:09:37,040
And you need to figure
that out as well.

225
00:09:37,040 --> 00:09:41,035
So just so you know that those
are those two user types.

226
00:09:41,035 --> 00:09:43,460
It's all user
testing, but having

227
00:09:43,460 --> 00:09:48,010
some granularity to find the
difference is a little useful.

228
00:09:48,010 --> 00:09:51,020
So the basics for technical
testing, which hopefully we

229
00:09:51,020 --> 00:09:54,040
can go through pretty quickly.

230
00:09:54,040 --> 00:09:56,830
I suspect that if you guys
have done programming,

231
00:09:56,830 --> 00:09:57,830
you've done this before.

232
00:09:57,830 --> 00:09:59,955
If you've had classes that
wanted code that worked,

233
00:09:59,955 --> 00:10:03,420
hopefully they've helped teach
you what the good things to do

234
00:10:03,420 --> 00:10:04,490
are.

235
00:10:04,490 --> 00:10:07,610
A lot of this sounds an
awful lot like good software

236
00:10:07,610 --> 00:10:10,160
practices lecture
that Andrew gave,

237
00:10:10,160 --> 00:10:13,440
because a lot of good testing
is, in fact, a good software

238
00:10:13,440 --> 00:10:14,190
practice.

239
00:10:14,190 --> 00:10:18,240
So have some
technical standards.

240
00:10:18,240 --> 00:10:19,350
Have a bug database.

241
00:10:19,350 --> 00:10:21,140
Or, given the size
of these projects,

242
00:10:21,140 --> 00:10:24,620
at least a bug Excel spreadsheet
with a list of everything

243
00:10:24,620 --> 00:10:26,320
that you've noticed.

244
00:10:26,320 --> 00:10:28,130
Build and test before
checking it in.

245
00:10:28,130 --> 00:10:29,110
Have daily builds.

246
00:10:29,110 --> 00:10:32,190
Have a daily play-through.

247
00:10:32,190 --> 00:10:33,910
Planned feature cuts.

248
00:10:33,910 --> 00:10:34,760
Scope.

249
00:10:34,760 --> 00:10:36,960
When you realize that you're
not getting everything

250
00:10:36,960 --> 00:10:41,550
done on Friday night as
you're working on your game,

251
00:10:41,550 --> 00:10:43,410
start cutting then.

252
00:10:43,410 --> 00:10:46,050
Don't wait until
Sunday night to decide

253
00:10:46,050 --> 00:10:47,540
what you're going to cut.

254
00:10:47,540 --> 00:10:50,980
Cut as soon as you realize that
you're blowing your estimates.

255
00:10:50,980 --> 00:10:53,380
That will allow you
to choose what you cut

256
00:10:53,380 --> 00:10:55,700
and to adjust
changes in your game

257
00:10:55,700 --> 00:10:57,980
to reflect the things you cut.

258
00:10:57,980 --> 00:11:00,260
It will also mean
that testing your game

259
00:11:00,260 --> 00:11:03,560
to make sure-- it will
introduce a lot fewer bugs

260
00:11:03,560 --> 00:11:06,280
when you do it in a controlled
manner than when you do it

261
00:11:06,280 --> 00:11:10,700
in desperation, which is
why I put it under testing.

262
00:11:10,700 --> 00:11:13,030
Finally, code freezes
and asset freezes.

263
00:11:13,030 --> 00:11:14,612
These are great,
and no one likes

264
00:11:14,612 --> 00:11:16,070
to use them because
it means you're

265
00:11:16,070 --> 00:11:20,060
going to waste an entire day
not working on your game.

266
00:11:20,060 --> 00:11:21,809
Who wants to do that?

267
00:11:21,809 --> 00:11:24,100
On the other hand, has anybody
here turned in a project

268
00:11:24,100 --> 00:11:26,750
that they've made a last
minute change to Sunday night,

269
00:11:26,750 --> 00:11:29,245
it was due Monday morning,
and then it didn't run at all?

270
00:11:32,472 --> 00:11:33,930
I got some honest
people out there.

271
00:11:33,930 --> 00:11:35,860
Good.

272
00:11:35,860 --> 00:11:38,255
The advantage of taking that
whole day to test your game

273
00:11:38,255 --> 00:11:42,926
is you turn something
in that works on Monday.

274
00:11:42,926 --> 00:11:44,300
Often, you turn
in something that

275
00:11:44,300 --> 00:11:46,217
works a lot better on
Monday because you've

276
00:11:46,217 --> 00:11:47,300
taken the time to test it.

277
00:11:47,300 --> 00:11:48,460
You've found the bugs.

278
00:11:48,460 --> 00:11:50,970
You've had a friend help you
review your code changes,

279
00:11:50,970 --> 00:11:52,760
and you really got it working.

280
00:11:52,760 --> 00:11:56,950
If you want to make a
good, polished, solid game,

281
00:11:56,950 --> 00:11:59,630
you will think about
it and give yourself

282
00:11:59,630 --> 00:12:02,440
a cushion for project two.

283
00:12:02,440 --> 00:12:03,760
That's probably maybe a day.

284
00:12:03,760 --> 00:12:04,760
It's a two week project.

285
00:12:04,760 --> 00:12:06,801
If you give yourself much
more cushion than that,

286
00:12:06,801 --> 00:12:08,600
you're not actually
working on it.

287
00:12:08,600 --> 00:12:11,270
But actively give
yourself that deadline

288
00:12:11,270 --> 00:12:13,930
and give yourself some
time to play your game.

289
00:12:18,030 --> 00:12:20,070
So that's the basics.

290
00:12:20,070 --> 00:12:22,370
Some advanced tips.

291
00:12:22,370 --> 00:12:23,610
It's all pretty simple.

292
00:12:23,610 --> 00:12:25,460
I think you've all
heard this all.

293
00:12:25,460 --> 00:12:28,084
How we maintain
your feature list.

294
00:12:28,084 --> 00:12:29,750
Agree on how features
are going to work,

295
00:12:29,750 --> 00:12:31,536
so talk with each other.

296
00:12:31,536 --> 00:12:33,285
Test those features
as soon as they go in.

297
00:12:36,064 --> 00:12:37,730
This may not be worth
it on these games,

298
00:12:37,730 --> 00:12:39,000
because they're small.

299
00:12:39,000 --> 00:12:40,860
When you hit your
project four game,

300
00:12:40,860 --> 00:12:42,840
you may want to
have one of these.

301
00:12:42,840 --> 00:12:45,420
Think about having a
standardized test plan that

302
00:12:45,420 --> 00:12:48,470
runs through all the buttons
and the branches and the pages

303
00:12:48,470 --> 00:12:50,890
and the options of your game.

304
00:12:50,890 --> 00:12:52,660
I realize that when
you've got randomness

305
00:12:52,660 --> 00:12:55,160
as one of your
core design goals,

306
00:12:55,160 --> 00:12:57,237
you can't predict how your
game is going to play.

307
00:12:57,237 --> 00:12:59,320
But you should be able to
take a look at your game

308
00:12:59,320 --> 00:13:01,900
and know most of the states
that it's going to get in.

309
00:13:01,900 --> 00:13:04,150
And you can create a
checklist that says, hey,

310
00:13:04,150 --> 00:13:06,275
have we exhausted
all the possibilities

311
00:13:06,275 --> 00:13:09,810
that a player will encounter?

312
00:13:09,810 --> 00:13:13,370
Have we tested them all in the
last two builds, yes or no?

313
00:13:13,370 --> 00:13:14,870
And the answer
should probably there

314
00:13:14,870 --> 00:13:18,202
be yes because one
of the things that's

315
00:13:18,202 --> 00:13:19,660
very common with
projects that have

316
00:13:19,660 --> 00:13:22,520
a lot of highly integrated
systems working together

317
00:13:22,520 --> 00:13:25,260
and depend on emergent
behavior-- which

318
00:13:25,260 --> 00:13:28,520
is most games-- is you can
introduce a feature over

319
00:13:28,520 --> 00:13:31,750
here that changes the way that
a feature that you implemented

320
00:13:31,750 --> 00:13:33,250
two weeks ago works.

321
00:13:33,250 --> 00:13:36,350
And you'll never know that it
did that until you go through

322
00:13:36,350 --> 00:13:38,930
and you test and you find that
something that was working

323
00:13:38,930 --> 00:13:40,530
is now broken.

324
00:13:40,530 --> 00:13:42,690
So it is worth it
to take the time

325
00:13:42,690 --> 00:13:44,370
to go back and
check those features

326
00:13:44,370 --> 00:13:45,760
you know nobody touched.

327
00:13:50,440 --> 00:13:51,610
Bug reporting.

328
00:13:51,610 --> 00:13:55,430
Because so many people
are so bad at this,

329
00:13:55,430 --> 00:13:58,100
I'm going to take the time
with a captured audience

330
00:13:58,100 --> 00:14:01,200
to tell you how to do it.

331
00:14:01,200 --> 00:14:02,680
Be kind to your
fellow engineers.

332
00:14:02,680 --> 00:14:05,570
Be kind to yourself, if you
end up getting your own bugs.

333
00:14:05,570 --> 00:14:09,190
Write good defect
reports and not bad ones.

334
00:14:09,190 --> 00:14:14,527
Good bugs include what happened,
what should have happened.

335
00:14:14,527 --> 00:14:16,110
Because it doesn't
do me a lot of good

336
00:14:16,110 --> 00:14:17,609
if you tell me that
the peach mobile

337
00:14:17,609 --> 00:14:19,310
is going 30 miles
per hour if I don't

338
00:14:19,310 --> 00:14:20,393
know why that's a problem.

339
00:14:23,240 --> 00:14:24,892
How do I reproduce it?

340
00:14:24,892 --> 00:14:27,350
If it's obvious to reproduce,
you don't need to include it.

341
00:14:27,350 --> 00:14:31,470
But if it took 38 steps,
include a text file.

342
00:14:31,470 --> 00:14:32,440
How common is the bug?

343
00:14:32,440 --> 00:14:33,850
How serious is the bug?

344
00:14:33,850 --> 00:14:36,480
Is this something that the team
needs to jump on right now,

345
00:14:36,480 --> 00:14:39,250
or is this something that
Sunday afternoon you're

346
00:14:39,250 --> 00:14:42,180
going to look at and
say, eh, ship it.

347
00:14:42,180 --> 00:14:45,159
Because some bugs you look
at and you say, eh, ship it.

348
00:14:45,159 --> 00:14:46,700
But you want to know
that it's there,

349
00:14:46,700 --> 00:14:48,480
and you want to
know how it's going

350
00:14:48,480 --> 00:14:53,385
to affect your game before
you just decide to ship it.

351
00:14:53,385 --> 00:14:55,760
Include some supporting data
if it's going to be helpful,

352
00:14:55,760 --> 00:14:56,540
and it often is.

353
00:14:56,540 --> 00:14:58,540
If you're trying to
describe something visually,

354
00:14:58,540 --> 00:15:00,620
give them a screenshot.

355
00:15:00,620 --> 00:15:02,160
If you can run it
in the debugger

356
00:15:02,160 --> 00:15:05,820
and catch some
debugging code, great.

357
00:15:05,820 --> 00:15:06,610
Save the files.

358
00:15:06,610 --> 00:15:08,930
Know the version of the
game you were using.

359
00:15:08,930 --> 00:15:10,650
Be ready to help
your team member who

360
00:15:10,650 --> 00:15:13,510
has to fix your horrible awful
crash that happens one out

361
00:15:13,510 --> 00:15:17,970
of every four times, and you're
not sure how you made it.

362
00:15:17,970 --> 00:15:19,920
Be willing to help
them recreate it,

363
00:15:19,920 --> 00:15:23,680
because it can be really
hard to recreate bugs.

364
00:15:23,680 --> 00:15:28,310
Finally, know the difference
between a criticism

365
00:15:28,310 --> 00:15:32,000
and a critique, and make
sure that your reports are

366
00:15:32,000 --> 00:15:35,050
critiques of the
game and the bugs

367
00:15:35,050 --> 00:15:38,610
and not criticisms of
your fellow developers

368
00:15:38,610 --> 00:15:40,080
and their ideas.

369
00:15:40,080 --> 00:15:42,420
And when I say criticism,
what I mean here

370
00:15:42,420 --> 00:15:44,915
is an unsupported
opinion usually delivered

371
00:15:44,915 --> 00:15:47,030
in an unprofessional tone.

372
00:15:47,030 --> 00:15:50,160
These purple elephants are
junk and it's totally stupid

373
00:15:50,160 --> 00:15:52,940
that they fly.

374
00:15:52,940 --> 00:15:55,270
A critique is a
statement of opinion

375
00:15:55,270 --> 00:15:59,400
about the game backed up by
examples and hopefully facts.

376
00:15:59,400 --> 00:16:02,280
The purple elephants
make the game too easy

377
00:16:02,280 --> 00:16:03,500
because they can fly.

378
00:16:03,500 --> 00:16:06,490
I can skip almost a
third of the game play

379
00:16:06,490 --> 00:16:08,330
in the bottom area
of the screen.

380
00:16:08,330 --> 00:16:09,250
There's a problem.

381
00:16:09,250 --> 00:16:10,810
There's a reason
why it's a problem.

382
00:16:10,810 --> 00:16:13,107
And there's a fact that
you can talk about,

383
00:16:13,107 --> 00:16:14,190
and you can have opinions.

384
00:16:14,190 --> 00:16:17,241
And it doesn't matter that the
purple elephants go in the game

385
00:16:17,241 --> 00:16:17,740
or not.

386
00:16:17,740 --> 00:16:20,170
You can just talk about what
they're doing in the game.

387
00:16:20,170 --> 00:16:21,961
You don't have to talk
about the person who

388
00:16:21,961 --> 00:16:24,670
put the purple elephants
in the game at all.

389
00:16:27,490 --> 00:16:29,080
So user testing.

390
00:16:29,080 --> 00:16:31,790
I think we've actually
covered this one to death.

391
00:16:31,790 --> 00:16:35,370
But if I did a
lecture about testing

392
00:16:35,370 --> 00:16:37,630
and didn't stop and talk
about why you're doing it,

393
00:16:37,630 --> 00:16:39,420
it would be a little odd.

394
00:16:39,420 --> 00:16:42,530
Once again, every time we
say iterative design, that

395
00:16:42,530 --> 00:16:45,720
means you make a design, you
test it, you get the data,

396
00:16:45,720 --> 00:16:48,300
you move on without actually
testing it with users.

397
00:16:48,300 --> 00:16:52,140
You don't get real data,
et cetera and so forth.

398
00:16:52,140 --> 00:16:54,840
So I think that drum has
been well and truly beaten.

399
00:16:54,840 --> 00:16:57,320
But here's a warning
we haven't really

400
00:16:57,320 --> 00:17:01,980
talked about yet, which
is test-driven design.

401
00:17:01,980 --> 00:17:05,960
Does not mean do what your
testers tell you to do.

402
00:17:05,960 --> 00:17:09,790
Test-driven design means
gather data from your testers,

403
00:17:09,790 --> 00:17:13,420
analyze it, and act on it.

404
00:17:13,420 --> 00:17:15,530
90% of all MIT
students will tell you

405
00:17:15,530 --> 00:17:18,440
that your game is too easy.

406
00:17:18,440 --> 00:17:20,490
They're wrong.

407
00:17:20,490 --> 00:17:23,000
90% of all people who like
to play first person shooters

408
00:17:23,000 --> 00:17:26,089
will tell you that
your game needs a gun.

409
00:17:26,089 --> 00:17:28,310
If you didn't plan on
having a gun in your game,

410
00:17:28,310 --> 00:17:32,350
it probably really
doesn't need a gun.

411
00:17:32,350 --> 00:17:34,722
Testers are often full of
solutions and suggestions

412
00:17:34,722 --> 00:17:36,430
to make your game just
like the game they

413
00:17:36,430 --> 00:17:37,554
wish they were playing.

414
00:17:37,554 --> 00:17:39,220
But you're not trying
to make that game.

415
00:17:39,220 --> 00:17:41,930
You're trying to make the
game you're trying to make.

416
00:17:41,930 --> 00:17:44,540
So when you gather
your data, put it

417
00:17:44,540 --> 00:17:47,270
through the filter of what
your vision of the game is

418
00:17:47,270 --> 00:17:49,250
and what you intend
your game to do,

419
00:17:49,250 --> 00:17:52,140
and don't let your
testers drive your game.

420
00:17:52,140 --> 00:17:57,240
Let your vision and
your data drive it.

421
00:17:57,240 --> 00:17:58,390
So how.

422
00:17:58,390 --> 00:17:59,952
How do you get
ideas from testers?

423
00:17:59,952 --> 00:18:02,285
And how do you actually turn
them into something useful?

424
00:18:05,327 --> 00:18:07,160
So you all remember the
experimental method.

425
00:18:07,160 --> 00:18:08,810
It's actually really
pretty similar.

426
00:18:08,810 --> 00:18:13,050
Going to form a hypothesis,
or ask a question.

427
00:18:13,050 --> 00:18:15,250
Figure out how to get the
data to that question,

428
00:18:15,250 --> 00:18:17,340
collect it, and analyze it.

429
00:18:17,340 --> 00:18:18,790
So forming a question.

430
00:18:18,790 --> 00:18:20,740
What do you need to
know about your game

431
00:18:20,740 --> 00:18:23,280
that your team can't answer?

432
00:18:23,280 --> 00:18:26,530
The more specific you
can be, the easier it is.

433
00:18:26,530 --> 00:18:28,450
Sometimes it is
hard to get specific

434
00:18:28,450 --> 00:18:29,340
because you've got
a game and you've

435
00:18:29,340 --> 00:18:30,740
got a whole lot of
interlinking systems,

436
00:18:30,740 --> 00:18:32,989
and it's hard to figure out
which one is making things

437
00:18:32,989 --> 00:18:33,870
not quite right.

438
00:18:33,870 --> 00:18:35,420
But do your best.

439
00:18:35,420 --> 00:18:37,310
In there a particular mechanic?

440
00:18:37,310 --> 00:18:39,650
Are people having a hard
time or an easy time

441
00:18:39,650 --> 00:18:41,732
getting started on your game?

442
00:18:41,732 --> 00:18:43,440
How engaged are people
in your game play?

443
00:18:43,440 --> 00:18:45,065
Are they playing the
game and grinning,

444
00:18:45,065 --> 00:18:47,844
or are they just sitting
there and hitting the keys?

445
00:18:47,844 --> 00:18:49,635
How many of your players
complete your game

446
00:18:49,635 --> 00:18:51,260
in about the time
you expected them to?

447
00:18:51,260 --> 00:18:56,870
How many of them complete
too quickly or take too long?

448
00:18:56,870 --> 00:18:59,150
So when you're running
scientific experiments,

449
00:18:59,150 --> 00:19:00,800
you try to standardize
your conditions

450
00:19:00,800 --> 00:19:02,860
as much as possible
to get data that can

451
00:19:02,860 --> 00:19:06,300
be compared with each other.

452
00:19:06,300 --> 00:19:09,080
When you're collecting
qualitative data, which

453
00:19:09,080 --> 00:19:10,980
is what most focus
testing data is,

454
00:19:10,980 --> 00:19:12,730
this is actually pretty
hard to do but you

455
00:19:12,730 --> 00:19:15,040
want to try to do it anyway.

456
00:19:15,040 --> 00:19:20,060
So some of the ways
you can do that is by

457
00:19:20,060 --> 00:19:22,760
make sure you're giving each
player the same set of starting

458
00:19:22,760 --> 00:19:25,716
information by having a
script to introduce your game,

459
00:19:25,716 --> 00:19:27,840
to give them any information
about the game they're

460
00:19:27,840 --> 00:19:30,600
going to need because your game
is in too early to have a UI.

461
00:19:30,600 --> 00:19:32,590
Games often don't
have UIs that are

462
00:19:32,590 --> 00:19:35,950
particularly useful while
they want to be tested early.

463
00:19:35,950 --> 00:19:39,780
That's OK, just tell your
players what your UI "should"

464
00:19:39,780 --> 00:19:41,414
tell them.

465
00:19:41,414 --> 00:19:43,080
You can double this
up with user testing

466
00:19:43,080 --> 00:19:45,970
by giving them pictures
of the intended UI,

467
00:19:45,970 --> 00:19:48,210
and you can see if they can
figure it out from there

468
00:19:48,210 --> 00:19:48,710
or not.

469
00:19:48,710 --> 00:19:51,730
But that's advanced
testing tricks.

470
00:19:51,730 --> 00:19:53,725
Have a dedicated observer.

471
00:19:53,725 --> 00:19:56,100
Don't depend on their memory
for what they're looking for

472
00:19:56,100 --> 00:19:58,060
or what they see.

473
00:19:58,060 --> 00:19:59,530
Give them a list
of specific things

474
00:19:59,530 --> 00:20:01,980
to look for that you think
will answer the questions

475
00:20:01,980 --> 00:20:03,290
you need to answer.

476
00:20:03,290 --> 00:20:05,500
Give them space to
write it down with.

477
00:20:05,500 --> 00:20:07,980
And give them the
autonomy to stand back

478
00:20:07,980 --> 00:20:10,270
and just take notes
while someone else

479
00:20:10,270 --> 00:20:15,520
offers your player assistance
or introduces them to the game.

480
00:20:15,520 --> 00:20:18,680
And finally, you really
can't observe everything.

481
00:20:18,680 --> 00:20:22,550
Think about what your players
might be able to tell you

482
00:20:22,550 --> 00:20:23,902
that you can't see.

483
00:20:23,902 --> 00:20:25,360
Write down a few
questions for them

484
00:20:25,360 --> 00:20:27,470
to answer afterwards
if you need to.

485
00:20:27,470 --> 00:20:28,370
Keep it short.

486
00:20:28,370 --> 00:20:30,578
You don't really want to
make somebody write an essay

487
00:20:30,578 --> 00:20:33,250
while they do your testing.

488
00:20:33,250 --> 00:20:36,145
If you can encourage your
players to talk to the game

489
00:20:36,145 --> 00:20:38,300
or to talk out loud
while they play,

490
00:20:38,300 --> 00:20:42,680
that's the greatest thing ever.

491
00:20:42,680 --> 00:20:44,920
The best people I've
ever heard play games

492
00:20:44,920 --> 00:20:50,120
are middle schoolers because
they connect their brain

493
00:20:50,120 --> 00:20:52,660
straight to their mouth
without any filter,

494
00:20:52,660 --> 00:20:53,685
and they're just there.

495
00:20:53,685 --> 00:20:55,310
They're telling you
what they're doing.

496
00:20:55,310 --> 00:20:56,490
They're telling you
why they're doing it.

497
00:20:56,490 --> 00:20:58,948
They're telling you why this
particular aspect of your game

498
00:20:58,948 --> 00:21:01,590
is the worst thing ever, or
why this particular aspect

499
00:21:01,590 --> 00:21:04,340
of your game is the
greatest thing ever.

500
00:21:04,340 --> 00:21:06,970
And you can just sit
there and listen to them

501
00:21:06,970 --> 00:21:08,920
and you know why
they're doing things.

502
00:21:08,920 --> 00:21:10,165
Adults are a lot quieter.

503
00:21:10,165 --> 00:21:12,540
We often don't want to explain
why we're doing something.

504
00:21:12,540 --> 00:21:15,210
Often it's because we don't
know we're supposed to be doing.

505
00:21:15,210 --> 00:21:17,420
And we don't want to admit that
I don't know what I'm doing.

506
00:21:17,420 --> 00:21:19,378
I'm just going to poke
stuff until I figure out

507
00:21:19,378 --> 00:21:21,580
what's going on.

508
00:21:21,580 --> 00:21:23,680
So that's where you
want to watch people.

509
00:21:23,680 --> 00:21:27,190
And if you really need
to, you can lead and say,

510
00:21:27,190 --> 00:21:29,160
why did you do that thing?

511
00:21:29,160 --> 00:21:32,432
Or even better, what
are you thinking?

512
00:21:32,432 --> 00:21:34,140
People will often
answer those questions.

513
00:21:34,140 --> 00:21:35,889
But you really want
to keep those prodding

514
00:21:35,889 --> 00:21:37,550
questions neutral.

515
00:21:37,550 --> 00:21:40,140
Don't say, why did you
do that crazy thing?

516
00:21:40,140 --> 00:21:43,160
Or, what are you
looking for back there?

517
00:21:43,160 --> 00:21:44,990
But just, what are you thinking?

518
00:21:44,990 --> 00:21:47,240
What are you doing?

519
00:21:47,240 --> 00:21:50,090
What's your plan?

520
00:21:50,090 --> 00:21:51,650
That sort of thing.

521
00:21:51,650 --> 00:21:53,670
Once you've got
your data the team

522
00:21:53,670 --> 00:21:57,470
can review it together to notice
trends, spot common problems,

523
00:21:57,470 --> 00:21:58,290
and so on.

524
00:21:58,290 --> 00:22:01,950
And that's where the things
that four or five or a majority

525
00:22:01,950 --> 00:22:05,555
of your testers did or
said or ran into, that's

526
00:22:05,555 --> 00:22:07,430
what you're going to be
pulling out and using

527
00:22:07,430 --> 00:22:10,030
to think about how you're
going to fix those problems.

528
00:22:10,030 --> 00:22:12,380
Or if you noticed a trend
that you hadn't noticed before

529
00:22:12,380 --> 00:22:14,250
that people were really
liking an aspect of your game

530
00:22:14,250 --> 00:22:15,990
you haven't thought
about, it's something

531
00:22:15,990 --> 00:22:18,300
you can work on magnifying
so that you can share it

532
00:22:18,300 --> 00:22:18,900
with everyone.

533
00:22:22,340 --> 00:22:26,230
So for those of you who fell
asleep while I was talking,

534
00:22:26,230 --> 00:22:28,860
the too long didn't
read version really

535
00:22:28,860 --> 00:22:32,250
is who are you testing with,
because your audience will

536
00:22:32,250 --> 00:22:33,920
affect your data.

537
00:22:33,920 --> 00:22:35,970
What are you testing with?

538
00:22:35,970 --> 00:22:38,770
The state of your game
will affect your data.

539
00:22:38,770 --> 00:22:41,952
That shouldn't stop you from
testing with early builds,

540
00:22:41,952 --> 00:22:44,160
but it does mean you need
to adjust your expectations

541
00:22:44,160 --> 00:22:46,450
and make sure your players
are ready to play with it.

542
00:22:49,370 --> 00:22:52,380
And finally, why
are you testing?

543
00:22:52,380 --> 00:22:54,490
Only you can answer
that question.

544
00:22:54,490 --> 00:22:57,284
We can give you a lot of
reasons or suggestions,

545
00:22:57,284 --> 00:22:58,700
but if you're
running a focus test

546
00:22:58,700 --> 00:23:02,210
it really needs to be because
you're looking for data

547
00:23:02,210 --> 00:23:05,110
that your team intends to use.

548
00:23:05,110 --> 00:23:07,240
And you can always use
data to support your game.

549
00:23:07,240 --> 00:23:07,950
You really can.

550
00:23:07,950 --> 00:23:10,460
Even if you don't realize
that you need it, you do.

551
00:23:13,360 --> 00:23:15,420
And I keep saying
"and gather data."

552
00:23:15,420 --> 00:23:16,990
And I talked about observations.

553
00:23:16,990 --> 00:23:19,080
I talked about surveys.

554
00:23:19,080 --> 00:23:23,300
I talked about interviews,
which are our three major tools

555
00:23:23,300 --> 00:23:28,264
unless you are going to go to
all the work of videotaping

556
00:23:28,264 --> 00:23:29,930
and audio recording
and getting releases

557
00:23:29,930 --> 00:23:32,030
from your testers, which
I really don't recommend.

558
00:23:32,030 --> 00:23:34,550
That's more work than
you should be doing.

559
00:23:34,550 --> 00:23:36,860
But the three major tools
really are observing people

560
00:23:36,860 --> 00:23:39,730
and taking notes, asking them
a standard set of questions

561
00:23:39,730 --> 00:23:42,700
afterwards-- either on
paper or in person--

562
00:23:42,700 --> 00:23:45,200
and thinking about
what kind of data

563
00:23:45,200 --> 00:23:50,260
you can gather best with
each of those methods.

564
00:23:50,260 --> 00:23:52,650
If you have 20 different
levels you want someone

565
00:23:52,650 --> 00:23:55,369
to rate at a one to five as
to whether it's good or not,

566
00:23:55,369 --> 00:23:57,035
you should just give
them piece of paper

567
00:23:57,035 --> 00:23:58,576
and when they finish
each level, they

568
00:23:58,576 --> 00:24:01,500
can check a number
between one and five.

569
00:24:01,500 --> 00:24:03,566
If you want to see how
long it takes someone

570
00:24:03,566 --> 00:24:05,440
to get started playing
the game, you probably

571
00:24:05,440 --> 00:24:07,700
just want to watch
them with a timer.

572
00:24:07,700 --> 00:24:10,140
How long does it take them
to get from that first screen

573
00:24:10,140 --> 00:24:14,820
to getting into the game?

574
00:24:14,820 --> 00:24:17,287
And finally, if you want to
get a general impression,

575
00:24:17,287 --> 00:24:19,370
sometimes talking to people
helps bring out things

576
00:24:19,370 --> 00:24:20,411
that you didn't think of.

577
00:24:20,411 --> 00:24:22,260
You can get some
insights you wouldn't

578
00:24:22,260 --> 00:24:24,970
have thought to ask for
by talking with people.

579
00:24:24,970 --> 00:24:27,219
But be a little careful
talking with people,

580
00:24:27,219 --> 00:24:28,760
because people don't
want to tell you

581
00:24:28,760 --> 00:24:30,710
bad things to your face.

582
00:24:30,710 --> 00:24:33,640
They'd rather write the bad
things down on a piece of paper

583
00:24:33,640 --> 00:24:35,514
where they don't have
to admit that they said

584
00:24:35,514 --> 00:24:36,970
bad things about this game.

585
00:24:36,970 --> 00:24:39,050
That covers less for
middle school students,

586
00:24:39,050 --> 00:24:42,260
as I mentioned earlier.

587
00:24:42,260 --> 00:24:44,450
But most people
don't really want

588
00:24:44,450 --> 00:24:46,860
to say mean things
to the person who

589
00:24:46,860 --> 00:24:50,040
made the game, because we don't
want to make people feel bad.

590
00:24:50,040 --> 00:24:51,710
So you need to
balance that out when

591
00:24:51,710 --> 00:24:53,460
you're asking people
questions about well,

592
00:24:53,460 --> 00:24:55,810
how did you like the game?

593
00:24:55,810 --> 00:24:58,171
The answer is usually,
oh, it's great.

594
00:24:58,171 --> 00:24:58,670
Yeah.

595
00:25:03,460 --> 00:25:07,010
So having run
through all of that,

596
00:25:07,010 --> 00:25:08,990
let me fight with my
computer and bring up

597
00:25:08,990 --> 00:25:09,905
a couple of examples.

598
00:25:13,630 --> 00:25:16,400
That's not the right form yet.

599
00:25:16,400 --> 00:25:19,015
So this is a script
created by students

600
00:25:19,015 --> 00:25:21,620
by a team that had a
dedicated QA lead person.

601
00:25:24,270 --> 00:25:27,220
I do not expect your sheets
to look anything like this,

602
00:25:27,220 --> 00:25:29,380
but I want you to see an
example of what someone

603
00:25:29,380 --> 00:25:32,430
did when it was their job
and they were taking time

604
00:25:32,430 --> 00:25:33,290
to get things ready.

605
00:25:33,290 --> 00:25:34,910
So you can see how
serious we are.

606
00:25:34,910 --> 00:25:38,110
And so you've got your
observer instructions.

607
00:25:38,110 --> 00:25:40,535
And what's day's goal?

608
00:25:40,535 --> 00:25:42,160
Finding out if the
feedback in the game

609
00:25:42,160 --> 00:25:45,540
works, along with a
couple of other things.

610
00:25:45,540 --> 00:25:47,900
And there's a testing
script written out

611
00:25:47,900 --> 00:25:51,170
pretty clearly along with
some basic instructions.

612
00:25:51,170 --> 00:25:56,054
Since these teams had a
dedicated focus test lead,

613
00:25:56,054 --> 00:25:57,470
they had someone
who'd spent a lot

614
00:25:57,470 --> 00:25:59,360
of time thinking about
testing and a lot of people

615
00:25:59,360 --> 00:26:01,360
who'd spend a lot of
time working on the game.

616
00:26:01,360 --> 00:26:03,730
So the lead tester
was in the position

617
00:26:03,730 --> 00:26:06,600
of trying to make sure
that the team understood

618
00:26:06,600 --> 00:26:09,920
how to run tests, because they
weren't experts at it either.

619
00:26:09,920 --> 00:26:13,290
So I just embedded all
of the good advice right

620
00:26:13,290 --> 00:26:16,260
into the script.

621
00:26:16,260 --> 00:26:17,290
And there you have it.

622
00:26:17,290 --> 00:26:19,400
And then there's a sheet
to write notes down.

623
00:26:22,740 --> 00:26:26,420
They've got the big questions
that they're worried about,

624
00:26:26,420 --> 00:26:27,410
so they can write down.

625
00:26:27,410 --> 00:26:28,826
They've got space
for other notes.

626
00:26:28,826 --> 00:26:32,240
And it means that whatever
they've got, they've got.

627
00:26:32,240 --> 00:26:34,590
And then finally, it follows
up with a fairly short

628
00:26:34,590 --> 00:26:38,680
questionnaire trying to get
some answers to questions

629
00:26:38,680 --> 00:26:41,926
they're wondering about.

630
00:26:41,926 --> 00:26:44,300
The specifics are always going
to vary from game to game.

631
00:26:44,300 --> 00:26:46,175
Looking at the focus
test sheet from one game

632
00:26:46,175 --> 00:26:48,000
isn't going to help
you wildly with yours.

633
00:26:48,000 --> 00:26:51,270
But you can think about the
way the questions are worded

634
00:26:51,270 --> 00:26:54,544
and the number of questions
so that you're not

635
00:26:54,544 --> 00:26:55,335
overloading people.

636
00:26:56,400 --> 00:26:59,050
I believe these are up
on Stellar as a sample

637
00:26:59,050 --> 00:27:03,500
if you want to download it
and look at it as an example.

638
00:27:03,500 --> 00:27:07,200
Finally, the paper
we expect you to do.

639
00:27:07,200 --> 00:27:11,990
So this is the focus
test report which

640
00:27:11,990 --> 00:27:13,540
is also download-able
on Stellar,

641
00:27:13,540 --> 00:27:17,500
I think in either Word or Rich
Text Format so you can use it.

642
00:27:17,500 --> 00:27:20,027
When we say we expect you to
turn in a focus test report,

643
00:27:20,027 --> 00:27:21,860
this is what we're
expecting you to turn in.

644
00:27:21,860 --> 00:27:24,799
If you take a look
at it, it walks you

645
00:27:24,799 --> 00:27:26,340
through the whole
process and there's

646
00:27:26,340 --> 00:27:27,770
a space for most of this stuff.

647
00:27:27,770 --> 00:27:31,430
So what's the goal
of the focus test?

648
00:27:31,430 --> 00:27:33,750
Record what happened
at the focus test

649
00:27:33,750 --> 00:27:36,770
and what are you going
to do afterwards.

650
00:27:36,770 --> 00:27:40,560
When you are planning your focus
test for this afternoon later

651
00:27:40,560 --> 00:27:44,070
in class, I recommend you
download this and use it

652
00:27:44,070 --> 00:27:45,640
as your sheet to
record notes on,

653
00:27:45,640 --> 00:27:50,390
because it'll help make sure
that you get the data that we

654
00:27:50,390 --> 00:27:53,760
are asking you to get.

655
00:27:53,760 --> 00:27:57,300
And I think that's
actually most of what

656
00:27:57,300 --> 00:27:58,610
I have to say about testing.

657
00:27:58,610 --> 00:27:59,609
Are there any questions?

658
00:28:03,159 --> 00:28:05,147
PHILLIP TAN: I have a question.

659
00:28:05,147 --> 00:28:09,620
What's currently using
some bug database,

660
00:28:09,620 --> 00:28:14,590
and if so, what are
folks currently using?

661
00:28:14,590 --> 00:28:17,572
Or let me rephrase that.

662
00:28:17,572 --> 00:28:20,057
Who has used a bug
database before?

663
00:28:20,057 --> 00:28:23,442
What are some of the ones
that you're familiar with?

664
00:28:23,442 --> 00:28:24,150
AUDIENCE: Chehra.

665
00:28:24,150 --> 00:28:26,898
PHILLIP TAN: Chehra?

666
00:28:26,898 --> 00:28:27,928
AUDIENCE: Bugzilla.

667
00:28:27,928 --> 00:28:28,886
PHILLIP TAN: Which one?

668
00:28:28,886 --> 00:28:29,383
AUDIENCE: Bugzilla.

669
00:28:29,383 --> 00:28:30,466
PHILLIP TAN: Oh, Bugzilla.

670
00:28:30,466 --> 00:28:31,371
Yeah.

671
00:28:31,371 --> 00:28:32,512
AUDIENCE: Buganizer.

672
00:28:32,512 --> 00:28:33,720
PHILLIP TAN: Which one is it?

673
00:28:33,720 --> 00:28:34,126
AUDIENCE: Buganizer.

674
00:28:34,126 --> 00:28:35,084
PHILLIP TAN: Buganizer?

675
00:28:35,084 --> 00:28:35,750
AUDIENCE: Yeah.

676
00:28:35,750 --> 00:28:37,234
PHILLIP TAN: Oh.

677
00:28:37,234 --> 00:28:38,472
AUDIENCE: I use [INAUDIBLE].

678
00:28:38,472 --> 00:28:39,222
PHILLIP TAN: Yeah.

679
00:28:39,222 --> 00:28:42,982
AUDIENCE: [INAUDIBLE] That's
the [INAUDIBLE] database, yeah.

680
00:28:42,982 --> 00:28:44,440
PHILLIP TAN: And
as Sara mentioned,

681
00:28:44,440 --> 00:28:47,671
you can also just use a Google
spreadsheet or something.

682
00:28:47,671 --> 00:28:51,647
AUDIENCE: [INAUDIBLE] is
a great bug database too.

683
00:28:51,647 --> 00:28:54,000
PHILLIP TAN: There is always
this intermediate step

684
00:28:54,000 --> 00:28:57,207
between someone entering
a bug into a bug database

685
00:28:57,207 --> 00:29:00,490
and then deciding that they're
going to fix it, right?

686
00:29:00,490 --> 00:29:02,872
So priority, somebody
actually says

687
00:29:02,872 --> 00:29:06,640
I think the bug that
the characters are

688
00:29:06,640 --> 00:29:09,142
too short or something,
and then the art director

689
00:29:09,142 --> 00:29:10,536
says no, not a bug.

690
00:29:10,536 --> 00:29:13,330
Will not fix that.

691
00:29:13,330 --> 00:29:17,338
So just because you're entering
something in a bug database

692
00:29:17,338 --> 00:29:19,462
doesn't necessarily
mean that your team's

693
00:29:19,462 --> 00:29:21,590
committed to actually
spending time on that.

694
00:29:21,590 --> 00:29:24,047
But when a team does decide
that, OK, this is a bug.

695
00:29:24,047 --> 00:29:25,463
Maybe because it
crashes the game.

696
00:29:25,463 --> 00:29:28,285
Maybe because it's
actually not how

697
00:29:28,285 --> 00:29:31,120
we want the game to perform.

698
00:29:31,120 --> 00:29:33,383
It needs to be entered
into your task list.

699
00:29:33,383 --> 00:29:36,040
So that means your
task list is changing

700
00:29:36,040 --> 00:29:40,100
and you need to be able
to put that in a priority.

701
00:29:40,100 --> 00:29:42,028
Possibly, this bug is
more serious than some

702
00:29:42,028 --> 00:29:44,438
of the planned tasks that
you already have on there.

703
00:29:44,438 --> 00:29:49,698
So you need to rearrange
that so that [INAUDIBLE].

704
00:29:49,698 --> 00:29:52,452
Something else that Sara
mentioned very briefly,

705
00:29:52,452 --> 00:29:54,660
but I want to
stress, MIT students

706
00:29:54,660 --> 00:29:57,943
are actually horrible testers.

707
00:29:57,943 --> 00:30:02,362
I've heard this many times from
many different companies here

708
00:30:02,362 --> 00:30:04,330
and [INAUDIBLE].

709
00:30:04,330 --> 00:30:08,120
MIT students are not a
representative population

710
00:30:08,120 --> 00:30:09,490
of the general public.

711
00:30:09,490 --> 00:30:11,270
You look at systems differently.

712
00:30:11,270 --> 00:30:13,720
You look at screens differently.

713
00:30:13,720 --> 00:30:15,660
So even though most
of your testing

714
00:30:15,660 --> 00:30:17,581
that we're going to be
doing this semester,

715
00:30:17,581 --> 00:30:20,284
I imagine that a lot of testing
you're going to do on your own

716
00:30:20,284 --> 00:30:24,362
is going to be with other people
in your dorms, for instance,

717
00:30:24,362 --> 00:30:26,692
and that's fine for this class.

718
00:30:26,692 --> 00:30:30,980
But don't think that it's normal
to use even university students

719
00:30:30,980 --> 00:30:32,525
as your only test audience.

720
00:30:32,525 --> 00:30:36,094
That's really not representative
of the general public.

721
00:30:36,094 --> 00:30:39,127
MIT students in particular.

722
00:30:39,127 --> 00:30:40,752
PROFESSOR: MIT students
do [INAUDIBLE].

723
00:30:40,752 --> 00:30:42,752
PHILLIP TAN: The final
thing, as Sara mentioned,

724
00:30:42,752 --> 00:30:44,300
user testing in two contexts.

725
00:30:44,300 --> 00:30:48,496
One for overall category of
you are testing with users

726
00:30:48,496 --> 00:30:51,570
and then it's broken down
to more specifically, focus

727
00:30:51,570 --> 00:30:53,016
testing and user testing.

728
00:30:53,016 --> 00:30:55,908
And you may be wondering
why user testing is used

729
00:30:55,908 --> 00:31:00,484
and that [INAUDIBLE] two
different things, as Sara said,

730
00:31:00,484 --> 00:31:02,480
it's good to be more granular.

731
00:31:02,480 --> 00:31:06,402
What we normally refer
to as usability testing,

732
00:31:06,402 --> 00:31:10,803
as Sara mentioned, has
only fairly recently

733
00:31:10,803 --> 00:31:12,656
become the standard name for it.

734
00:31:12,656 --> 00:31:13,280
PROFESSOR: Yes.

735
00:31:13,280 --> 00:31:13,780
Yeah.

736
00:31:13,780 --> 00:31:17,567
PHILLIP TAN: Back in the '90s
and even the early 2000s,

737
00:31:17,567 --> 00:31:18,983
user testing and
usability testing

738
00:31:18,983 --> 00:31:21,415
were basically the
same thing, because we

739
00:31:21,415 --> 00:31:23,730
were talking about websites
and enterprise software.

740
00:31:23,730 --> 00:31:25,142
We were not talking about games.

741
00:31:25,142 --> 00:31:27,350
So the only kind of user
testing that you'd be doing,

742
00:31:27,350 --> 00:31:29,540
putting a piece of software
in front of someone,

743
00:31:29,540 --> 00:31:31,350
getting feedback, was
usability testing.

744
00:31:31,350 --> 00:31:34,315
That's why they had the
same name for so long.

745
00:31:34,315 --> 00:31:36,970
But now there's focus testing,
which is very, very applicable

746
00:31:36,970 --> 00:31:40,523
not just for games but also for
things like user experience.

747
00:31:40,523 --> 00:31:43,290
I had not heard that term
user experience [INAUDIBLE].

748
00:31:43,290 --> 00:31:46,744
So where it's not just can the
player figure out what to do,

749
00:31:46,744 --> 00:31:49,510
but the person actually
having the desired experience

750
00:31:49,510 --> 00:31:50,545
for playing the game.

751
00:31:50,545 --> 00:31:55,230
That's why now there's
multiple names for that.

752
00:31:55,230 --> 00:31:57,710
PROFESSOR: Thank you.

753
00:31:57,710 --> 00:31:59,150
Then I think I'm out of here.

754
00:31:59,150 --> 00:32:02,095
I'm going to sneak
off with my computer.

755
00:32:02,095 --> 00:32:03,220
GENEVIEVE CONLEY: Hi, guys.

756
00:32:03,220 --> 00:32:04,710
My name is Genevieve Conley.

757
00:32:04,710 --> 00:32:07,020
I'm a researcher at Riot.

758
00:32:07,020 --> 00:32:08,680
And some other
terms that you might

759
00:32:08,680 --> 00:32:11,760
have heard for that, user
experience researcher, user

760
00:32:11,760 --> 00:32:14,380
researcher, or the way we
were just talking about it,

761
00:32:14,380 --> 00:32:16,405
like user tester basically.

762
00:32:16,405 --> 00:32:17,580
And a little bit about me.

763
00:32:17,580 --> 00:32:20,210
I graduated from MIT in 2010.

764
00:32:20,210 --> 00:32:22,520
I got my degree in Course 9.

765
00:32:22,520 --> 00:32:25,810
And I actually worked
at the MIT Gambit game

766
00:32:25,810 --> 00:32:27,520
lab on a couple games there.

767
00:32:27,520 --> 00:32:30,010
So that's actually where I got
my start doing user testing,

768
00:32:30,010 --> 00:32:32,700
was in the summer projects
we would run play tests

769
00:32:32,700 --> 00:32:34,760
with local community
members, have them come in,

770
00:32:34,760 --> 00:32:39,010
and that's where I actually
did my first user tests.

771
00:32:39,010 --> 00:32:40,990
So I work at Riot Games.

772
00:32:40,990 --> 00:32:44,140
How many of you guys have heard
of League of Legends before?

773
00:32:44,140 --> 00:32:44,640
OK.

774
00:32:44,640 --> 00:32:47,510
How many of you actively
play League of Legends?

775
00:32:47,510 --> 00:32:48,050
OK, good.

776
00:32:48,050 --> 00:32:50,050
So I'll give a little bit
of context on the game

777
00:32:50,050 --> 00:32:51,090
just so I'm not losing anybody.

778
00:32:51,090 --> 00:32:52,290
And I'll try to
make sure I explain

779
00:32:52,290 --> 00:32:54,870
all of our jargon, because
sometimes when you're in a game

780
00:32:54,870 --> 00:32:57,745
company you just start using
the language like everybody

781
00:32:57,745 --> 00:32:58,620
speaks that language.

782
00:32:58,620 --> 00:33:01,440
But a little bit about Riot
is we make League of Legends.

783
00:33:01,440 --> 00:33:03,750
And our core mission
is that we aspire

784
00:33:03,750 --> 00:33:06,590
to be the most player-focused
game company in the world.

785
00:33:06,590 --> 00:33:08,290
And this is going to
be relevant to what

786
00:33:08,290 --> 00:33:10,331
I'm talking about today,
because you can't really

787
00:33:10,331 --> 00:33:13,080
be player-focused if you
don't talk to your players.

788
00:33:13,080 --> 00:33:15,360
So what this means at
Riot is we think about all

789
00:33:15,360 --> 00:33:18,370
of our decisions in terms of
how much player value it adds,

790
00:33:18,370 --> 00:33:21,749
or how much player pain
something negative causes.

791
00:33:21,749 --> 00:33:24,040
So this is like-- I don't
know if any of you familiar--

792
00:33:24,040 --> 00:33:25,722
we have an eSports
scene, and right now

793
00:33:25,722 --> 00:33:27,680
we're in the middle of
our world championships.

794
00:33:27,680 --> 00:33:30,500
And we created a video and
a song that would basically,

795
00:33:30,500 --> 00:33:33,630
we hoped to be the anthem
of the world championship.

796
00:33:33,630 --> 00:33:36,100
And this is something that a
couple people at the studio

797
00:33:36,100 --> 00:33:38,360
thought was really important, so
they just went out and made it.

798
00:33:38,360 --> 00:33:40,526
And the point was just to
make the players pumped up

799
00:33:40,526 --> 00:33:42,370
about worlds and to
give them a song that

800
00:33:42,370 --> 00:33:43,575
represented how they feel.

801
00:33:43,575 --> 00:33:45,450
And this is just a small
example of something

802
00:33:45,450 --> 00:33:47,286
that happens at
Riot where we really

803
00:33:47,286 --> 00:33:48,910
just want to do
something that we think

804
00:33:48,910 --> 00:33:50,380
players will think is cool.

805
00:33:50,380 --> 00:33:52,030
And one of the
ways if we do that

806
00:33:52,030 --> 00:33:53,760
is through play testing
and user testing.

807
00:33:53,760 --> 00:33:55,680
And I'm going to use a little
bit different terminology,

808
00:33:55,680 --> 00:33:58,000
so I'll make sure that I
tie it back into the way

809
00:33:58,000 --> 00:34:00,190
that Sara's been talking
about it as we go.

810
00:34:00,190 --> 00:34:01,690
So a little about
League of Legends.

811
00:34:01,690 --> 00:34:04,290
It's generally a 5v5 game.

812
00:34:04,290 --> 00:34:05,670
It's a team oriented game.

813
00:34:05,670 --> 00:34:07,570
So you play as a
member of a team.

814
00:34:07,570 --> 00:34:10,190
And you usually play a specific
role or position on that team,

815
00:34:10,190 --> 00:34:12,139
just like basketball.

816
00:34:12,139 --> 00:34:15,029
And you play as one
of many champions.

817
00:34:15,029 --> 00:34:16,570
And it's online and
it's competitive.

818
00:34:16,570 --> 00:34:18,760
So like I said, we have
an eSports scene where

819
00:34:18,760 --> 00:34:20,060
you play against another team.

820
00:34:20,060 --> 00:34:23,086
And you go basically across this
map, try to take objectives,

821
00:34:23,086 --> 00:34:24,460
and ultimately
blow up this thing

822
00:34:24,460 --> 00:34:26,168
called the Nexus which
is a giant crystal

823
00:34:26,168 --> 00:34:29,469
on the other side of the map.

824
00:34:29,469 --> 00:34:31,254
And you play as a champion.

825
00:34:31,254 --> 00:34:33,370
And these are player
characters that each

826
00:34:33,370 --> 00:34:36,909
have an individual set of
skills, different play styles,

827
00:34:36,909 --> 00:34:39,360
and there's over 120
of them in our game.

828
00:34:39,360 --> 00:34:41,989
So even though each
game is session based--

829
00:34:41,989 --> 00:34:44,667
you don't level up your
character necessarily

830
00:34:44,667 --> 00:34:46,250
over the lifetime
of playing the game,

831
00:34:46,250 --> 00:34:49,159
each session is a new session--
there's lots of different ways

832
00:34:49,159 --> 00:34:51,150
to play because we have lots
of different characters, which

833
00:34:51,150 --> 00:34:52,358
I'll talk to in a little bit.

834
00:34:55,250 --> 00:34:56,750
So one of things
that I talked about

835
00:34:56,750 --> 00:34:59,290
is it's really hard to
be player-focused if you

836
00:34:59,290 --> 00:35:01,330
don't talk to your players.

837
00:35:01,330 --> 00:35:03,000
And one of things at
Riot is that we all

838
00:35:03,000 --> 00:35:03,990
play League of Legends.

839
00:35:03,990 --> 00:35:05,700
In fact, some of
us play it a lot.

840
00:35:05,700 --> 00:35:07,480
Some of us play a lot at work.

841
00:35:07,480 --> 00:35:09,640
But just because you
play your own game

842
00:35:09,640 --> 00:35:12,250
doesn't mean that you
actually know what it's

843
00:35:12,250 --> 00:35:13,530
like for all of your players.

844
00:35:13,530 --> 00:35:16,140
And using MIT students
is a great example.

845
00:35:16,140 --> 00:35:17,970
If you guys just play
with MIT students,

846
00:35:17,970 --> 00:35:20,130
you might not know what
it's like for people

847
00:35:20,130 --> 00:35:22,380
in other countries, for example.

848
00:35:22,380 --> 00:35:24,130
So one of things we
try to do is make sure

849
00:35:24,130 --> 00:35:25,860
that we understand
our players broadly

850
00:35:25,860 --> 00:35:27,390
and know that we
are our players,

851
00:35:27,390 --> 00:35:31,510
but we are a small
sub-segment of our players.

852
00:35:31,510 --> 00:35:33,300
The way we use
research is it equips

853
00:35:33,300 --> 00:35:36,090
us to understand player beliefs,
their needs, their desires,

854
00:35:36,090 --> 00:35:37,590
and their relationship
with the game

855
00:35:37,590 --> 00:35:39,870
and with other
people in the game.

856
00:35:39,870 --> 00:35:43,170
And what this allows us to
do is design experiences.

857
00:35:43,170 --> 00:35:46,764
So there's specific
experiences, interactions

858
00:35:46,764 --> 00:35:47,930
that we're hoping to design.

859
00:35:47,930 --> 00:35:49,890
So in general when we're
creating a feature,

860
00:35:49,890 --> 00:35:52,550
we have a set of goals that
we're trying to achieve.

861
00:35:52,550 --> 00:35:55,590
And what we want to do is
we want to help designers.

862
00:35:55,590 --> 00:35:58,110
developers, QA,
artists, we want to help

863
00:35:58,110 --> 00:35:59,489
them achieve these goals.

864
00:35:59,489 --> 00:36:02,030
So we want to know exactly what
they're trying to do and then

865
00:36:02,030 --> 00:36:04,240
get them feedback on how
well those goals are being

866
00:36:04,240 --> 00:36:06,360
communicated to players,
how well they're actually

867
00:36:06,360 --> 00:36:08,110
able to interact with
those, and then what

868
00:36:08,110 --> 00:36:11,036
their reception is as a result.

869
00:36:11,036 --> 00:36:12,660
So the way I'm going
to break this down

870
00:36:12,660 --> 00:36:14,850
is play tests and user
labs, which is essentially

871
00:36:14,850 --> 00:36:16,516
the way we were talking
about it before,

872
00:36:16,516 --> 00:36:19,740
is play tests and user tests.

873
00:36:19,740 --> 00:36:23,500
So play test is the backbone
of creating your game.

874
00:36:23,500 --> 00:36:25,900
It's really, really
difficult to create a game

875
00:36:25,900 --> 00:36:27,040
without play testing it.

876
00:36:27,040 --> 00:36:29,540
So I'm glad that's really part
of how you guys are designing

877
00:36:29,540 --> 00:36:30,410
your games.

878
00:36:30,410 --> 00:36:32,820
And play tests basically
allow us to quickly gather

879
00:36:32,820 --> 00:36:35,140
information about
a game or a feature

880
00:36:35,140 --> 00:36:36,750
to see how it's working.

881
00:36:36,750 --> 00:36:39,036
And typically, this
is done with resources

882
00:36:39,036 --> 00:36:40,160
that are readily available.

883
00:36:40,160 --> 00:36:43,470
So people in your dorms, people
in your class, or at Riot,

884
00:36:43,470 --> 00:36:47,212
we often do it with other
Rioters, and even on the team.

885
00:36:47,212 --> 00:36:49,170
And some of the benefits
we can get out of this

886
00:36:49,170 --> 00:36:51,060
is we can identify
technical issues.

887
00:36:51,060 --> 00:36:53,067
Now, Sara talked
about a specific type

888
00:36:53,067 --> 00:36:55,150
of this, which is quality
assurance testing, which

889
00:36:55,150 --> 00:36:56,446
is less what I'm talking about.

890
00:36:56,446 --> 00:36:58,320
I'm more talking about
testing with your team

891
00:36:58,320 --> 00:37:00,710
to just quickly identify
technical issues that

892
00:37:00,710 --> 00:37:02,722
might prevent you from
playing that game.

893
00:37:02,722 --> 00:37:04,430
You can also look at
things like balance.

894
00:37:04,430 --> 00:37:06,710
So in a competitive game
like League of Legends,

895
00:37:06,710 --> 00:37:08,550
you play against other players.

896
00:37:08,550 --> 00:37:11,160
And if one type of character
is really overpowered

897
00:37:11,160 --> 00:37:13,950
or has a significant advantage
over another type of character,

898
00:37:13,950 --> 00:37:16,140
we can quickly identify
that through play testing.

899
00:37:16,140 --> 00:37:18,850
So for instance, our designers
have two play tests a day

900
00:37:18,850 --> 00:37:20,970
where they test their
in-progress champions

901
00:37:20,970 --> 00:37:23,150
to see if there's anything
basically game breaking

902
00:37:23,150 --> 00:37:25,970
like that that they can quickly
identify and iterate on.

903
00:37:25,970 --> 00:37:27,489
We can also look
at things for fun.

904
00:37:27,489 --> 00:37:29,030
I mean, this is the
most obvious one.

905
00:37:29,030 --> 00:37:31,196
But it's really important
when you're making a game,

906
00:37:31,196 --> 00:37:33,840
if the game is supposed to be
fun, that it is actually fun.

907
00:37:33,840 --> 00:37:36,400
And in our case, we do
want to have a fun game.

908
00:37:36,400 --> 00:37:39,250
So we have a whole group
of designers and developers

909
00:37:39,250 --> 00:37:41,300
that have a team which
is known as the play team

910
00:37:41,300 --> 00:37:43,190
and they make
temporary game modes.

911
00:37:43,190 --> 00:37:45,010
So I don't know if
anybody played recently,

912
00:37:45,010 --> 00:37:47,430
but we had a big in-game
event called Shurima,

913
00:37:47,430 --> 00:37:49,410
and we had this game
mode called Ascension.

914
00:37:49,410 --> 00:37:50,868
And when they were
developing that,

915
00:37:50,868 --> 00:37:54,070
they were often several times
a day-- many, many times a day,

916
00:37:54,070 --> 00:37:56,296
in some cases--
testing every iteration

917
00:37:56,296 --> 00:37:58,170
that they made to that
game mode to make sure

918
00:37:58,170 --> 00:37:59,242
that it was actually fun.

919
00:37:59,242 --> 00:38:00,700
And a lot of times
what you'll find

920
00:38:00,700 --> 00:38:03,490
is that your game isn't
necessarily fun sometimes.

921
00:38:03,490 --> 00:38:05,730
And that's OK, because each
time you make a change,

922
00:38:05,730 --> 00:38:07,410
you're getting closer
to that fun experience.

923
00:38:07,410 --> 00:38:09,640
One day, you'll be playing
your game and like, oh my gosh,

924
00:38:09,640 --> 00:38:10,570
it's actually fun now.

925
00:38:10,570 --> 00:38:11,153
This is great.

926
00:38:11,153 --> 00:38:11,800
Let's do this.

927
00:38:11,800 --> 00:38:14,641
So it's a really good
way to get that feedback.

928
00:38:14,641 --> 00:38:16,890
And another specific area
that we're looking for a lot

929
00:38:16,890 --> 00:38:18,305
when we play test is clarity.

930
00:38:18,305 --> 00:38:19,680
And clarity is a
word that we use

931
00:38:19,680 --> 00:38:22,500
to mean basically how
clear is the information--

932
00:38:22,500 --> 00:38:25,590
either the visual information
or the mechanical information--

933
00:38:25,590 --> 00:38:27,820
how well is that being
communicated to players?

934
00:38:27,820 --> 00:38:29,680
So this could be
something like with UI,

935
00:38:29,680 --> 00:38:31,430
if you just have a
picture and you show it

936
00:38:31,430 --> 00:38:34,850
to players or to your
dorm-mates, your classmates,

937
00:38:34,850 --> 00:38:38,040
how easily are they able to
identify what a button means?

938
00:38:38,040 --> 00:38:40,475
Or which things are
clickable and not clickable.

939
00:38:40,475 --> 00:38:42,350
Those things are all
really, really important

940
00:38:42,350 --> 00:38:44,150
in a game like
ours where we have

941
00:38:44,150 --> 00:38:46,935
five people on a team, 10
people on the map, each

942
00:38:46,935 --> 00:38:49,060
with their own set of
abilities and visual effects.

943
00:38:49,060 --> 00:38:50,810
And there could be a
lot of visual clutter

944
00:38:50,810 --> 00:38:53,079
if we didn't think about
this type of thing.

945
00:38:53,079 --> 00:38:55,620
And then the way we typically
get these, at least at Riot, is

946
00:38:55,620 --> 00:38:56,720
like I said, team tests.

947
00:38:56,720 --> 00:38:59,260
That's the example I used with
the designers and the play

948
00:38:59,260 --> 00:38:59,761
team.

949
00:38:59,761 --> 00:39:01,343
They're just playing
within their team

950
00:39:01,343 --> 00:39:03,680
frequently to make sure that
they can quickly and easily

951
00:39:03,680 --> 00:39:05,300
give each other feedback.

952
00:39:05,300 --> 00:39:06,960
Quality assurance
test is another type

953
00:39:06,960 --> 00:39:09,000
that's a very specific
implementation of this,

954
00:39:09,000 --> 00:39:10,420
which Sara spoke to.

955
00:39:10,420 --> 00:39:12,456
And in house "guerrilla
tests"-- and I always

956
00:39:12,456 --> 00:39:14,330
have to be careful how
I say "guerrilla," not

957
00:39:14,330 --> 00:39:15,960
"gorilla." "Guerrilla."

958
00:39:15,960 --> 00:39:17,950
This is when we bring
other Rioters who

959
00:39:17,950 --> 00:39:20,670
aren't part of a team who
maybe haven't had exposure

960
00:39:20,670 --> 00:39:22,090
to that particular feature.

961
00:39:22,090 --> 00:39:23,790
This is particularly
useful for things

962
00:39:23,790 --> 00:39:27,400
like clarity or usability tests,
because we don't necessarily

963
00:39:27,400 --> 00:39:30,590
need to have that perception
or that sentiment that

964
00:39:30,590 --> 00:39:32,450
is really unique to players.

965
00:39:32,450 --> 00:39:34,500
What we need here
instead is just see, hey,

966
00:39:34,500 --> 00:39:37,637
how easily is this usable for
a reasonably tech savvy person?

967
00:39:37,637 --> 00:39:40,220
But even then, of course, there
are some cultural differences.

968
00:39:40,220 --> 00:39:41,760
There's some
hardware differences.

969
00:39:41,760 --> 00:39:44,510
So it's a first phase when you
do them as guerrilla testing.

970
00:39:44,510 --> 00:39:46,635
And you always want to make
sure that you follow up

971
00:39:46,635 --> 00:39:49,800
with real players
after the fact.

972
00:39:49,800 --> 00:39:52,280
So user labs, this is
what I really focus in.

973
00:39:52,280 --> 00:39:54,154
So I use a lot of
different methods at Riot,

974
00:39:54,154 --> 00:39:55,570
but one of the key
ways that we're

975
00:39:55,570 --> 00:39:58,500
able to get player feedback
is by bringing players

976
00:39:58,500 --> 00:40:00,577
to Riot to have
them play the game.

977
00:40:00,577 --> 00:40:02,160
So this is a way to
help us understand

978
00:40:02,160 --> 00:40:06,430
how real players will
actually approach a feature.

979
00:40:06,430 --> 00:40:09,650
What we get out of this is
really, really important.

980
00:40:09,650 --> 00:40:11,130
We get reception and interest.

981
00:40:11,130 --> 00:40:13,640
Like, how stoked are players
going to be about something,

982
00:40:13,640 --> 00:40:16,420
or what we call the
table flip reaction.

983
00:40:16,420 --> 00:40:18,422
Which is like, we're
working on an update

984
00:40:18,422 --> 00:40:19,630
to Summoner's Rift right now.

985
00:40:19,630 --> 00:40:20,736
And it was early on.

986
00:40:20,736 --> 00:40:22,360
It was really important
for us to know,

987
00:40:22,360 --> 00:40:23,770
were players going
to be really upset

988
00:40:23,770 --> 00:40:26,230
that we were changing something
that's so key to the game.

989
00:40:26,230 --> 00:40:27,920
So we made sure to
loop them on very early

990
00:40:27,920 --> 00:40:30,420
in on the process to see how
they were responding to changes

991
00:40:30,420 --> 00:40:31,700
we were going to make.

992
00:40:31,700 --> 00:40:34,220
We also look for things like
usage patterns and errors.

993
00:40:34,220 --> 00:40:35,595
And this is
important that we get

994
00:40:35,595 --> 00:40:37,740
a broad swath of
different types of players

995
00:40:37,740 --> 00:40:39,750
to look at this type
of thing, especially

996
00:40:39,750 --> 00:40:42,070
with things like we
just updated the client.

997
00:40:42,070 --> 00:40:44,300
Making sure that the changes
we made to the interface

998
00:40:44,300 --> 00:40:46,180
weren't going to
confuse players--

999
00:40:46,180 --> 00:40:48,810
experienced players who
were used to our old version

1000
00:40:48,810 --> 00:40:50,280
and new players
who would be coming

1001
00:40:50,280 --> 00:40:51,770
into our game for
the first time.

1002
00:40:51,770 --> 00:40:54,100
We also look at
needs and desires.

1003
00:40:54,100 --> 00:40:58,350
And this is not like Maslov's--
Maslow's hierarchy of needs.

1004
00:40:58,350 --> 00:41:00,750
It's more like, what are
things that players really

1005
00:41:00,750 --> 00:41:02,840
want from our game?

1006
00:41:02,840 --> 00:41:05,870
And understanding that
can be difficult sometimes

1007
00:41:05,870 --> 00:41:07,950
because, like I said,
we all play our game.

1008
00:41:07,950 --> 00:41:09,700
We're very entrenched
in this experience.

1009
00:41:09,700 --> 00:41:11,650
We have our own
experience in the game

1010
00:41:11,650 --> 00:41:13,560
that we feel very
strongly about.

1011
00:41:13,560 --> 00:41:15,706
But understanding that
other players might

1012
00:41:15,706 --> 00:41:17,080
have different
needs and desires,

1013
00:41:17,080 --> 00:41:18,663
and understanding
what proportion that

1014
00:41:18,663 --> 00:41:20,980
represents within our
broader player base,

1015
00:41:20,980 --> 00:41:23,430
these are really important
things to understand.

1016
00:41:23,430 --> 00:41:25,320
How we get this is a
lot of different ways

1017
00:41:25,320 --> 00:41:26,790
that I'm actually going
to go into in a little bit

1018
00:41:26,790 --> 00:41:27,660
more detail.

1019
00:41:27,660 --> 00:41:30,147
And each one has a
specific application,

1020
00:41:30,147 --> 00:41:31,730
but there are a few
that we use pretty

1021
00:41:31,730 --> 00:41:33,230
much in every user
lab, which I'm

1022
00:41:33,230 --> 00:41:35,202
going to go through right now.

1023
00:41:35,202 --> 00:41:37,160
So the first is something
we just talked about,

1024
00:41:37,160 --> 00:41:38,455
which is just observation.

1025
00:41:38,455 --> 00:41:41,080
And I bring this up because it's
something that anybody can do.

1026
00:41:41,080 --> 00:41:44,000
You don't need a dedicated
researcher like myself.

1027
00:41:44,000 --> 00:41:46,210
You guys should and
will be doing this.

1028
00:41:46,210 --> 00:41:48,200
And so in this
case, I have Chris,

1029
00:41:48,200 --> 00:41:50,860
who is a developer at Riot,
sitting in for a player.

1030
00:41:50,860 --> 00:41:53,550
And I have [? Jisan, ?] who
is another researcher at Riot,

1031
00:41:53,550 --> 00:41:56,310
sitting over his shoulder
watching him play.

1032
00:41:56,310 --> 00:41:58,570
In our case, we typically
just have one researcher

1033
00:41:58,570 --> 00:42:00,400
in the room both
taking the notes

1034
00:42:00,400 --> 00:42:02,140
and doing the
observation, just because

1035
00:42:02,140 --> 00:42:03,690
of resource limitations.

1036
00:42:03,690 --> 00:42:06,420
And because we've done this a
lot and it's our full time job.

1037
00:42:06,420 --> 00:42:09,220
So in this case,
[? Jisan ?] is watching

1038
00:42:09,220 --> 00:42:11,012
Chris interact with the client.

1039
00:42:11,012 --> 00:42:13,470
And you can see in his hand,
he's actually got the script--

1040
00:42:13,470 --> 00:42:15,670
or what we call
protocol-- in his hands

1041
00:42:15,670 --> 00:42:18,160
so he can remember what things
he should be looking for,

1042
00:42:18,160 --> 00:42:19,392
what things he wants to say.

1043
00:42:19,392 --> 00:42:21,850
In this case, we don't generally
try to use it as a script.

1044
00:42:21,850 --> 00:42:24,082
We try to use it as
a memory refresher.

1045
00:42:24,082 --> 00:42:26,290
That way, players can feel
really comfortable with us

1046
00:42:26,290 --> 00:42:27,873
and feel like they
have a relationship

1047
00:42:27,873 --> 00:42:30,692
and can have a
comfortable conversation.

1048
00:42:30,692 --> 00:42:32,650
One of the ways we achieve
this is through what

1049
00:42:32,650 --> 00:42:34,945
we call the Think Aloud
protocol, which is what Sara

1050
00:42:34,945 --> 00:42:36,350
was referencing,
which is basically we

1051
00:42:36,350 --> 00:42:38,350
try to make sure that
they have this direct link

1052
00:42:38,350 --> 00:42:39,740
from their mind to their mouth.

1053
00:42:39,740 --> 00:42:41,530
And there's a couple
ways to achieve this

1054
00:42:41,530 --> 00:42:43,590
that I'll speak to,
but the goal of this

1055
00:42:43,590 --> 00:42:46,150
is we want to know exactly
what the game is communicating

1056
00:42:46,150 --> 00:42:48,950
to players, and then how
players are processing that

1057
00:42:48,950 --> 00:42:50,492
and creating an internal model.

1058
00:42:50,492 --> 00:42:51,950
And the best way
to do that is just

1059
00:42:51,950 --> 00:42:53,610
to step back and
let them tell you

1060
00:42:53,610 --> 00:42:55,210
exactly what they're seeing.

1061
00:42:55,210 --> 00:42:57,270
And if you aren't quite
sure what they're seeing,

1062
00:42:57,270 --> 00:42:59,542
then you can follow up, which
I'll get to in a second.

1063
00:42:59,542 --> 00:43:02,000
So here's an example of a player
who's seeing our character

1064
00:43:02,000 --> 00:43:04,210
Lucian, who is a
champion who was released

1065
00:43:04,210 --> 00:43:05,810
fairly recently in the game.

1066
00:43:05,810 --> 00:43:08,190
She's seeing him before
he had been released

1067
00:43:08,190 --> 00:43:09,940
in a work in progress
state and giving us

1068
00:43:09,940 --> 00:43:12,374
feedback on what she sees.

1069
00:43:12,374 --> 00:43:13,040
[VIDEO PLAYBACK]

1070
00:43:13,040 --> 00:43:17,140
-I can see that he walks
faster than the normal--

1071
00:43:17,140 --> 00:43:18,012
[END PLAYBACK]

1072
00:43:18,012 --> 00:43:21,002
GENEVIEVE CONLEY: Sorry, we
forgot to plug in the audio.

1073
00:43:21,002 --> 00:43:22,710
I think you guys could
hear it all right.

1074
00:43:26,066 --> 00:43:26,566
All right.

1075
00:43:26,566 --> 00:43:27,048
[VIDEO PLAYBACK]

1076
00:43:27,048 --> 00:43:27,530
-[INAUDIBLE].

1077
00:43:27,530 --> 00:43:28,488
GENEVIEVE CONLEY: Oops.

1078
00:43:31,386 --> 00:43:36,832
-I can see that he walks faster
than the normal character

1079
00:43:36,832 --> 00:43:41,188
would, but his animation
doesn't make it seem so.

1080
00:43:41,188 --> 00:43:44,576
It seems like he takes
a really big stride,

1081
00:43:44,576 --> 00:43:49,076
but I was imagining him
to walk really fast.

1082
00:43:49,076 --> 00:43:51,556
But it seems like it
looks like that way.

1083
00:43:51,556 --> 00:43:52,140
[END PLAYBACK]

1084
00:43:52,140 --> 00:43:53,764
GENEVIEVE CONLEY: So
this is an example

1085
00:43:53,764 --> 00:43:56,210
where the player has just
brought this up to us that she

1086
00:43:56,210 --> 00:43:58,650
expected him to move one
way, and he's actually

1087
00:43:58,650 --> 00:44:00,349
operating in a different way.

1088
00:44:00,349 --> 00:44:02,140
You can think of this
is as like, basically

1089
00:44:02,140 --> 00:44:03,750
a violation of
expectations, which

1090
00:44:03,750 --> 00:44:07,010
is often what we're looking for
especially with our characters.

1091
00:44:07,010 --> 00:44:09,580
Does what you see and what you
hear about a champion match

1092
00:44:09,580 --> 00:44:11,340
with how they actually play?

1093
00:44:11,340 --> 00:44:13,540
So one way you can do this
is to just let them talk.

1094
00:44:13,540 --> 00:44:17,090
And it is really important
not to be giving feedback on

1095
00:44:17,090 --> 00:44:19,150
whether or not you
think they're correct,

1096
00:44:19,150 --> 00:44:20,840
especially if they
do have a question.

1097
00:44:20,840 --> 00:44:23,300
It's also really important
not to be judgmental, right?

1098
00:44:23,300 --> 00:44:24,895
So the way you phrase
your questions,

1099
00:44:24,895 --> 00:44:26,770
even if you aren't
actually being judgmental,

1100
00:44:26,770 --> 00:44:29,620
sometimes players might think
that you're judging them.

1101
00:44:29,620 --> 00:44:32,120
So one of the contexts
we give players upfront

1102
00:44:32,120 --> 00:44:33,942
is always for us,
since we actually

1103
00:44:33,942 --> 00:44:36,400
didn't work on the game or the
feature that they're seeing,

1104
00:44:36,400 --> 00:44:38,540
we can say, we
didn't work on this.

1105
00:44:38,540 --> 00:44:40,920
It's really important that
you give us honest feedback.

1106
00:44:40,920 --> 00:44:42,669
I'm not going to go
home and cry at night,

1107
00:44:42,669 --> 00:44:44,894
so I want you to give me
totally honest feedback.

1108
00:44:44,894 --> 00:44:46,810
And we also let them
know that sometimes we're

1109
00:44:46,810 --> 00:44:49,060
going to ask them questions,
and we're asking it

1110
00:44:49,060 --> 00:44:50,670
so that we can
understand what they're

1111
00:44:50,670 --> 00:44:51,920
thinking because
that's what we're

1112
00:44:51,920 --> 00:44:53,086
interested in understanding.

1113
00:44:53,086 --> 00:44:54,480
We're not asking
it because there

1114
00:44:54,480 --> 00:44:56,966
is a right or wrong answer.

1115
00:44:56,966 --> 00:44:58,590
So one of the ways
that makes it easier

1116
00:44:58,590 --> 00:45:00,910
to make sure that
we understand what

1117
00:45:00,910 --> 00:45:02,455
their mental model
of the game is

1118
00:45:02,455 --> 00:45:04,600
is a follow-up with
some probing questions.

1119
00:45:04,600 --> 00:45:06,330
So the player in
the previous example

1120
00:45:06,330 --> 00:45:08,220
was pretty clear about
what she was seeing

1121
00:45:08,220 --> 00:45:10,066
and why it violated
her expectations,

1122
00:45:10,066 --> 00:45:11,690
but sometimes players
don't necessarily

1123
00:45:11,690 --> 00:45:14,110
know why it doesn't actually
fit what they expected.

1124
00:45:14,110 --> 00:45:16,110
Or they might know that
something's frustrating,

1125
00:45:16,110 --> 00:45:18,340
but they're not necessarily
giving you the answer why.

1126
00:45:18,340 --> 00:45:19,560
So one of the ways
you can follow it up

1127
00:45:19,560 --> 00:45:21,170
is either with probing
questions, which

1128
00:45:21,170 --> 00:45:23,640
you ask in the moment, or you
can follow up with interviews

1129
00:45:23,640 --> 00:45:24,500
after the fact.

1130
00:45:24,500 --> 00:45:26,499
So here's an example of
[? Jisan ?] following up

1131
00:45:26,499 --> 00:45:29,214
with a player who just saw
Lucian for the first time.

1132
00:45:29,214 --> 00:45:29,880
[VIDEO PLAYBACK]

1133
00:45:29,880 --> 00:45:32,130
-How was that play [INAUDIBLE]?

1134
00:45:32,130 --> 00:45:33,250
-I thought it was fun.

1135
00:45:33,250 --> 00:45:35,896
I thought it was
[? different. ?]

1136
00:45:35,896 --> 00:45:38,782
-What makes it fun and
something different about it?

1137
00:45:42,680 --> 00:45:47,914
-So he, I guess, normal
[INAUDIBLE] skills,

1138
00:45:47,914 --> 00:45:49,122
just increased the intensity.

1139
00:45:51,800 --> 00:45:55,290
A lot of his skills based off
of move speed and [INAUDIBLE]

1140
00:45:55,290 --> 00:45:58,720
speed, and I guess he
feels [? confident. ?]

1141
00:45:58,720 --> 00:45:59,454
[END PLAYBACK]

1142
00:45:59,454 --> 00:46:01,120
GENEVIEVE CONLEY: So
this is an example.

1143
00:46:01,120 --> 00:46:02,578
A lot of times
you'll find this, is

1144
00:46:02,578 --> 00:46:04,530
that when you ask
that initial question,

1145
00:46:04,530 --> 00:46:06,260
you get a short response.

1146
00:46:06,260 --> 00:46:09,010
And what we're always trying
to do as researchers is invite

1147
00:46:09,010 --> 00:46:12,260
a conversation where they're
just really comfortable telling

1148
00:46:12,260 --> 00:46:14,855
us exactly what they feel and
giving us a lot of detail.

1149
00:46:14,855 --> 00:46:17,230
So in the first question, he
asked a really broad opening

1150
00:46:17,230 --> 00:46:19,200
question, which
was "how was that?"

1151
00:46:19,200 --> 00:46:20,992
He didn't even ask,
"how did that feel,"

1152
00:46:20,992 --> 00:46:22,700
which might lead in
a certain direction--

1153
00:46:22,700 --> 00:46:24,930
although that would be an
appropriate question as well.

1154
00:46:24,930 --> 00:46:26,400
He left it wide
open for the player

1155
00:46:26,400 --> 00:46:28,502
to take us where they
wanted to talk about it.

1156
00:46:28,502 --> 00:46:29,960
So the player
mentioned that it was

1157
00:46:29,960 --> 00:46:31,500
fun and something different.

1158
00:46:31,500 --> 00:46:33,430
So [? Jisan ?] wanted
to get more depth

1159
00:46:33,430 --> 00:46:35,940
into what does that actually
mean, because that alone

1160
00:46:35,940 --> 00:46:37,560
doesn't give us
enough direction to be

1161
00:46:37,560 --> 00:46:40,020
able to provide to our
designers to say what was good,

1162
00:46:40,020 --> 00:46:41,130
what was working.

1163
00:46:41,130 --> 00:46:44,300
So he mirrored exactly what
the player said back at him.

1164
00:46:44,300 --> 00:46:46,600
He said, oh, what
was fun and what

1165
00:46:46,600 --> 00:46:48,540
was the thing that
made it feel different?

1166
00:46:48,540 --> 00:46:50,540
And then the player, that
way we're not actually

1167
00:46:50,540 --> 00:46:52,130
putting our own
spin on interpreting

1168
00:46:52,130 --> 00:46:54,746
what we think they're saying is
fun or maybe what's different.

1169
00:46:54,746 --> 00:46:56,120
We're just mirroring
exactly back

1170
00:46:56,120 --> 00:46:59,075
what they said so that they
can give us more detail.

1171
00:46:59,075 --> 00:47:00,450
You have to be
careful with that,

1172
00:47:00,450 --> 00:47:02,320
because you also don't want
to sound cold and clinical

1173
00:47:02,320 --> 00:47:04,770
like you're sitting in an
office somewhere writing down

1174
00:47:04,770 --> 00:47:06,495
everything they say,
which you will be.

1175
00:47:06,495 --> 00:47:09,371
You want to make it feel like
it's a natural conversation.

1176
00:47:09,371 --> 00:47:10,870
So we're really
lucky because we get

1177
00:47:10,870 --> 00:47:12,440
to work with League
of Legends players

1178
00:47:12,440 --> 00:47:14,390
so we already have a
connection built up with them

1179
00:47:14,390 --> 00:47:16,973
where we can talk about it, but
it's really important that you

1180
00:47:16,973 --> 00:47:19,290
make that as hey, I want
to know what you think.

1181
00:47:19,290 --> 00:47:20,374
You're really valuable.

1182
00:47:20,374 --> 00:47:22,040
Let's just have a
conversation about it.

1183
00:47:22,040 --> 00:47:25,750
And it can be really
challenging sometimes.

1184
00:47:25,750 --> 00:47:27,530
So this is one that
you guys are already

1185
00:47:27,530 --> 00:47:30,120
pretty familiar with,
too, is part of interviews

1186
00:47:30,120 --> 00:47:33,259
that I've clumped in here is
basically the questionnaire.

1187
00:47:33,259 --> 00:47:34,800
Sara gave you guys
some great context

1188
00:47:34,800 --> 00:47:36,242
about when this might be useful.

1189
00:47:36,242 --> 00:47:37,700
Here's an example
of when we use it

1190
00:47:37,700 --> 00:47:39,690
in League of Legends research.

1191
00:47:39,690 --> 00:47:41,830
So this is an example
with Azir, who

1192
00:47:41,830 --> 00:47:44,536
is a character who just came
out in League of Legends.

1193
00:47:44,536 --> 00:47:46,160
This is actually from
quite a long time

1194
00:47:46,160 --> 00:47:47,991
ago when he was in
the concept phase.

1195
00:47:47,991 --> 00:47:49,490
And this is something
we do actually

1196
00:47:49,490 --> 00:47:52,031
with most of our concepts, is
we put them in front of players

1197
00:47:52,031 --> 00:47:54,140
before they go into
development to see, hey,

1198
00:47:54,140 --> 00:47:55,170
is this something
that feels like it

1199
00:47:55,170 --> 00:47:56,794
fits in the world of
League of Legends?

1200
00:47:56,794 --> 00:47:58,940
Is it clear where maybe
the source of power is?

1201
00:47:58,940 --> 00:48:01,148
And does this look like a
champion you would actually

1202
00:48:01,148 --> 00:48:01,760
want to play?

1203
00:48:01,760 --> 00:48:04,446
Now, we don't use data to
drive decisions at Riot.

1204
00:48:04,446 --> 00:48:05,820
I want to make
that really clear.

1205
00:48:05,820 --> 00:48:06,810
We use it to inform.

1206
00:48:06,810 --> 00:48:08,440
So just because something
wasn't resonating

1207
00:48:08,440 --> 00:48:09,815
with players in
our labs wouldn't

1208
00:48:09,815 --> 00:48:11,610
mean that we would
cancel a character.

1209
00:48:11,610 --> 00:48:13,220
It's just a way for
us to get feedback

1210
00:48:13,220 --> 00:48:15,480
for our designers who
already have their experience

1211
00:48:15,480 --> 00:48:18,320
and the experience
testing with other Rioters

1212
00:48:18,320 --> 00:48:20,815
to help them make
informed decisions.

1213
00:48:20,815 --> 00:48:22,940
So what's useful about a
questionnaire in this case

1214
00:48:22,940 --> 00:48:26,200
is that we can compare
across champions over time

1215
00:48:26,200 --> 00:48:28,270
and across different
roles or positions

1216
00:48:28,270 --> 00:48:32,334
and see what is resonating
and what does good look like.

1217
00:48:32,334 --> 00:48:34,000
We also use this as
a jumping off point.

1218
00:48:34,000 --> 00:48:35,260
So we start with
a questionnaire,

1219
00:48:35,260 --> 00:48:37,545
and then we go into a more
detailed interview about it

1220
00:48:37,545 --> 00:48:39,670
to make sure that we're
getting enough detail to be

1221
00:48:39,670 --> 00:48:41,836
able to provide useful
information to the designers.

1222
00:48:44,140 --> 00:48:46,090
So you guys are also
probably pretty familiar

1223
00:48:46,090 --> 00:48:48,290
with the idea of
experiments in general.

1224
00:48:48,290 --> 00:48:51,550
But part of research testing
can be devising experiments.

1225
00:48:51,550 --> 00:48:54,420
So we often hire people
from psychology backgrounds

1226
00:48:54,420 --> 00:48:56,010
or research
backgrounds in academia

1227
00:48:56,010 --> 00:48:58,580
because they have experience
setting up experiments.

1228
00:48:58,580 --> 00:49:00,100
So this is like the classic one.

1229
00:49:00,100 --> 00:49:01,390
I don't know if you guys
have already covered

1230
00:49:01,390 --> 00:49:02,780
this, the Engelbart experiment.

1231
00:49:02,780 --> 00:49:03,450
OK.

1232
00:49:03,450 --> 00:49:06,020
So this is foundational
human computer interaction.

1233
00:49:06,020 --> 00:49:08,430
It's one of the
most cited studies.

1234
00:49:08,430 --> 00:49:11,290
And it's English, Engerlbart,
and Berman basically

1235
00:49:11,290 --> 00:49:14,450
were looking for a
way to improve a way

1236
00:49:14,450 --> 00:49:17,310
to interact with a screen
or text manipulation.

1237
00:49:17,310 --> 00:49:18,997
And what they
proposed was something

1238
00:49:18,997 --> 00:49:20,830
that was radical and
was called the "mouse."

1239
00:49:20,830 --> 00:49:22,714
Maybe you've heard of it.

1240
00:49:22,714 --> 00:49:24,130
So that thing in
the right, I know

1241
00:49:24,130 --> 00:49:25,960
it's not a really
clear photo, but that

1242
00:49:25,960 --> 00:49:28,842
is one of the first early
prototypes of a mouse.

1243
00:49:28,842 --> 00:49:30,800
And what they wanted to
do is they want to see,

1244
00:49:30,800 --> 00:49:33,160
is this better than some
of these other interactions

1245
00:49:33,160 --> 00:49:35,380
out there.

1246
00:49:35,380 --> 00:49:38,380
Qualitative experience alone
might not be a very strong way

1247
00:49:38,380 --> 00:49:40,330
to prove it in this
case, especially when you

1248
00:49:40,330 --> 00:49:43,160
can collect quantitative
data to back it up.

1249
00:49:43,160 --> 00:49:46,400
So what they did is
they used the time

1250
00:49:46,400 --> 00:49:48,860
that it took to complete a
task, a very specific task that

1251
00:49:48,860 --> 00:49:51,235
was given across all these
different ways of manipulating

1252
00:49:51,235 --> 00:49:52,040
text.

1253
00:49:52,040 --> 00:49:55,305
And they saw how many errors
the user made in this case.

1254
00:49:55,305 --> 00:49:57,180
So you can see that the
mouse did pretty well

1255
00:49:57,180 --> 00:49:58,221
in terms of time overall.

1256
00:49:58,221 --> 00:50:00,430
It didn't take the
participants very long

1257
00:50:00,430 --> 00:50:02,180
to be able to complete the task.

1258
00:50:02,180 --> 00:50:03,760
But it had very
few errors, which

1259
00:50:03,760 --> 00:50:06,940
made it overall the best
out of these different ways

1260
00:50:06,940 --> 00:50:07,740
to select text.

1261
00:50:07,740 --> 00:50:09,656
And you can see some of
the other ones on here

1262
00:50:09,656 --> 00:50:13,080
were things like knee control,
joystick, a graphicon, which

1263
00:50:13,080 --> 00:50:17,170
is like an arc drawer, a
light pen, and a joystick.

1264
00:50:17,170 --> 00:50:19,020
And all of these were
ways that we could

1265
00:50:19,020 --> 00:50:21,050
have gone instead of the mouse.

1266
00:50:21,050 --> 00:50:22,540
But based on some
of this research,

1267
00:50:22,540 --> 00:50:24,147
they were able to
improve their design

1268
00:50:24,147 --> 00:50:25,980
and now you can see I'm
using one right now.

1269
00:50:29,630 --> 00:50:32,124
And as part of the experimental
approach, I broke it out,

1270
00:50:32,124 --> 00:50:33,540
but there's a
specific way you can

1271
00:50:33,540 --> 00:50:36,020
look at it, which is using
physiological techniques.

1272
00:50:36,020 --> 00:50:38,720
And there are many, and I'm
just going to cover two.

1273
00:50:38,720 --> 00:50:41,270
And typically, these are
used for very specific types

1274
00:50:41,270 --> 00:50:42,020
of research.

1275
00:50:42,020 --> 00:50:43,000
They're very exciting.

1276
00:50:43,000 --> 00:50:45,666
So oftentimes, you'll see people
want to use them on everything.

1277
00:50:45,666 --> 00:50:49,020
But they're very particularly
useful for certain cases.

1278
00:50:49,020 --> 00:50:52,240
So this is an example of eye
tracking, which is basically

1279
00:50:52,240 --> 00:50:54,140
fundamentally, we
put a camera up

1280
00:50:54,140 --> 00:50:56,432
that's able to specially
track where eye movements are.

1281
00:50:56,432 --> 00:50:58,931
And there are more sophisticated
ways you can look at this--

1282
00:50:58,931 --> 00:51:00,692
and I'm sure a bunch
of researchers at MIT

1283
00:51:00,692 --> 00:51:02,150
could tell you a
lot more about it.

1284
00:51:02,150 --> 00:51:03,510
But in this case, it
was pretty simple.

1285
00:51:03,510 --> 00:51:05,551
We were just looking at
where people were looking

1286
00:51:05,551 --> 00:51:08,330
on the screen and how
often and how long

1287
00:51:08,330 --> 00:51:09,430
they were looking there.

1288
00:51:09,430 --> 00:51:11,120
So you can see the
red parts of the map

1289
00:51:11,120 --> 00:51:14,870
here are the parts where there
are more intense gazes or more

1290
00:51:14,870 --> 00:51:16,520
gazes overall.

1291
00:51:16,520 --> 00:51:18,112
So unsurprisingly
to probably most

1292
00:51:18,112 --> 00:51:19,820
of the League of
Legends players in here,

1293
00:51:19,820 --> 00:51:21,840
or you guys as
gamers in general,

1294
00:51:21,840 --> 00:51:24,250
the mini map has a lot of gaze.

1295
00:51:24,250 --> 00:51:27,250
The bottom part, which is where
the abilities are displayed,

1296
00:51:27,250 --> 00:51:28,510
also has a lot of gaze.

1297
00:51:28,510 --> 00:51:30,010
That has a lot of
information that's

1298
00:51:30,010 --> 00:51:32,601
really relevant to players
like the cool down timer,

1299
00:51:32,601 --> 00:51:34,850
basically when the ability
is going to be ready again.

1300
00:51:34,850 --> 00:51:37,183
And the center of the screen
is where most of the action

1301
00:51:37,183 --> 00:51:37,890
happens.

1302
00:51:37,890 --> 00:51:40,251
So this type of information
is helpful for us.

1303
00:51:40,251 --> 00:51:42,500
If we want to, for instance,
introduce new information

1304
00:51:42,500 --> 00:51:43,970
on the screen,
we'd want to know,

1305
00:51:43,970 --> 00:51:45,230
hey, are we going
to introduce this

1306
00:51:45,230 --> 00:51:47,700
in a place that's going to get
a lot of traffic naturally.

1307
00:51:47,700 --> 00:51:49,400
But also is it going
to interrupt things

1308
00:51:49,400 --> 00:51:52,667
that that gaze is there for
a very important reason,

1309
00:51:52,667 --> 00:51:54,250
and maybe we shouldn't
interrupt that.

1310
00:51:54,250 --> 00:51:58,162
So we can help us to inform
those types of decisions.

1311
00:51:58,162 --> 00:52:00,120
There's also many other
types of physiological,

1312
00:52:00,120 --> 00:52:02,350
and I'll just hint at
one right now which

1313
00:52:02,350 --> 00:52:05,760
is things like galvanic
skin response, anything that

1314
00:52:05,760 --> 00:52:08,600
has to do with
physiological arousal,

1315
00:52:08,600 --> 00:52:10,790
either for stress or excitement.

1316
00:52:10,790 --> 00:52:13,250
These can help with situations
where maybe players wouldn't

1317
00:52:13,250 --> 00:52:15,380
be able to vocalize
why something

1318
00:52:15,380 --> 00:52:17,710
was frustrating or exciting.

1319
00:52:17,710 --> 00:52:19,840
Or it can also help
combat some biases

1320
00:52:19,840 --> 00:52:21,980
like recency
effect, where if you

1321
00:52:21,980 --> 00:52:23,790
ask a player at
the end of the lab

1322
00:52:23,790 --> 00:52:26,014
or at the end of a play
test what was interesting

1323
00:52:26,014 --> 00:52:27,930
or what was frustrating,
they might preference

1324
00:52:27,930 --> 00:52:29,440
things that were at the end.

1325
00:52:29,440 --> 00:52:32,540
Or they might preference things
that were really emotionally

1326
00:52:32,540 --> 00:52:35,400
significant for them, but might
forget smaller things that

1327
00:52:35,400 --> 00:52:38,580
are still maybe small
frustrating things that

1328
00:52:38,580 --> 00:52:41,431
build up over time.

1329
00:52:41,431 --> 00:52:43,180
So none of this is
really that interesting

1330
00:52:43,180 --> 00:52:45,820
if I don't give you an example
of how we actually do at Riot.

1331
00:52:45,820 --> 00:52:47,000
So here's a case study.

1332
00:52:47,000 --> 00:52:50,859
And I actually presented this
last year as part of a talk

1333
00:52:50,859 --> 00:52:51,650
that we did at MIT.

1334
00:52:51,650 --> 00:52:54,760
So I apologize if a little
bit is re-run for you guys.

1335
00:52:54,760 --> 00:52:57,531
I'll try to spice it up
a little bit as I go.

1336
00:52:57,531 --> 00:52:59,905
So I'm going to be talking
about Lucian, who I introduced

1337
00:52:59,905 --> 00:53:00,920
to you guys earlier.

1338
00:53:00,920 --> 00:53:03,250
He is one of the player
characters or champions

1339
00:53:03,250 --> 00:53:06,760
in League of Legends.

1340
00:53:06,760 --> 00:53:08,594
So where we always
start as researchers,

1341
00:53:08,594 --> 00:53:10,760
and part of the job that I
haven't really alluded to

1342
00:53:10,760 --> 00:53:14,070
yet is we spend a lot of time
talking to our designers,

1343
00:53:14,070 --> 00:53:16,480
to our developers, to our
artists, to our producers,

1344
00:53:16,480 --> 00:53:18,027
to make sure we
understand the goals

1345
00:53:18,027 --> 00:53:19,110
they're trying to achieve.

1346
00:53:19,110 --> 00:53:20,110
And I don't know
how many of you guys

1347
00:53:20,110 --> 00:53:22,460
got to catch August
Browning's talk last night,

1348
00:53:22,460 --> 00:53:24,030
but for the champion
designers, they

1349
00:53:24,030 --> 00:53:26,867
start with a very specific
idea of what goals

1350
00:53:26,867 --> 00:53:27,950
they're trying to achieve.

1351
00:53:27,950 --> 00:53:29,980
What is this player
character going to feel like?

1352
00:53:29,980 --> 00:53:32,021
What are you going to feel
like when you play it?

1353
00:53:32,021 --> 00:53:34,130
What is the mechanics
supposed to feel like?

1354
00:53:34,130 --> 00:53:35,870
What is the theme
supposed to feel like?

1355
00:53:35,870 --> 00:53:37,150
How does this
resonate with players?

1356
00:53:37,150 --> 00:53:38,350
How does it fit in
League of Legends?

1357
00:53:38,350 --> 00:53:40,391
These are all really
important questions for them

1358
00:53:40,391 --> 00:53:43,081
that they're constantly
asking themselves.

1359
00:53:43,081 --> 00:53:44,830
And in the case of
Lucian, he was supposed

1360
00:53:44,830 --> 00:53:47,250
to feel fast, sleek, speedy.

1361
00:53:47,250 --> 00:53:48,440
He was like a gun ninja.

1362
00:53:48,440 --> 00:53:49,841
Like pew, pew.

1363
00:53:49,841 --> 00:53:50,340
No?

1364
00:53:50,340 --> 00:53:51,900
All right.

1365
00:53:51,900 --> 00:53:53,950
Trying to see if you
guys were awake there.

1366
00:53:53,950 --> 00:53:56,080
And he's a range
damage character

1367
00:53:56,080 --> 00:53:57,480
with finesse game play.

1368
00:53:57,480 --> 00:54:02,280
And he basically has
the high skill ceiling.

1369
00:54:02,280 --> 00:54:05,220
And what that means is basically
he's harder to pick up,

1370
00:54:05,220 --> 00:54:08,370
but when you master him it's
a really rewarding feeling.

1371
00:54:08,370 --> 00:54:11,250
So that's a trade off
between accessibility, maybe

1372
00:54:11,250 --> 00:54:12,860
how quickly a player
can pick him up,

1373
00:54:12,860 --> 00:54:14,810
or what level a player
can pick him up,

1374
00:54:14,810 --> 00:54:17,050
for a reward of how
rewarding that feels when

1375
00:54:17,050 --> 00:54:19,180
you do master that champion.

1376
00:54:19,180 --> 00:54:22,407
And some of the challenges in
this were, this guy, his theme

1377
00:54:22,407 --> 00:54:23,240
is pretty different.

1378
00:54:23,240 --> 00:54:26,480
His equilibrium style, if
you've ever seen that movie.

1379
00:54:26,480 --> 00:54:28,480
And that's not something
we had necessarily

1380
00:54:28,480 --> 00:54:30,271
done that much in League
of Legends before,

1381
00:54:30,271 --> 00:54:32,510
so we wanted to make
sure, does he even fit.

1382
00:54:32,510 --> 00:54:34,135
And we also wanted
to know how do

1383
00:54:34,135 --> 00:54:37,210
you let a player feel super
fast and speedy in what

1384
00:54:37,210 --> 00:54:39,230
is essentially an
RTS style game.

1385
00:54:39,230 --> 00:54:42,580
It's a challenge.

1386
00:54:42,580 --> 00:54:45,680
So this is what the test
plan eventually looked like.

1387
00:54:45,680 --> 00:54:47,990
It's not necessarily
where we started from.

1388
00:54:47,990 --> 00:54:49,685
But we started with
concept testing,

1389
00:54:49,685 --> 00:54:51,810
and then we went through
actually three lab studies

1390
00:54:51,810 --> 00:54:53,780
with iteration in between
based on the feedback

1391
00:54:53,780 --> 00:54:55,110
we got from players.

1392
00:54:55,110 --> 00:54:59,990
And after he launched, we did
some post-launch evaluation.

1393
00:54:59,990 --> 00:55:01,880
So step one, just like
I showed you earlier,

1394
00:55:01,880 --> 00:55:03,210
was to do this concept testing.

1395
00:55:03,210 --> 00:55:07,850
And we got some pretty good
feedback based on that.

1396
00:55:07,850 --> 00:55:09,620
His fantasy was
good with players.

1397
00:55:09,620 --> 00:55:10,520
They liked the mood.

1398
00:55:10,520 --> 00:55:10,850
[THUD]

1399
00:55:10,850 --> 00:55:11,770
They felt like he fit.

1400
00:55:11,770 --> 00:55:12,793
Are you all right?

1401
00:55:12,793 --> 00:55:14,482
AUDIENCE: Yeah.

1402
00:55:14,482 --> 00:55:15,940
GENEVIEVE CONLEY:
And he made sense

1403
00:55:15,940 --> 00:55:18,340
for the role of an AD carry,
which again is like a range

1404
00:55:18,340 --> 00:55:20,180
damage dealer in this case.

1405
00:55:20,180 --> 00:55:23,120
So he had high appeal scores and
the theme felt really strong,

1406
00:55:23,120 --> 00:55:25,250
but one thing we very
quickly identified--

1407
00:55:25,250 --> 00:55:26,874
it's a little hard
to see on this one--

1408
00:55:26,874 --> 00:55:28,660
but his guns were
different colors.

1409
00:55:28,660 --> 00:55:31,060
So players expected the
guns to do different things

1410
00:55:31,060 --> 00:55:33,670
based on their colors, which
was not what we were planning.

1411
00:55:33,670 --> 00:55:35,170
So this was very
good early feedback

1412
00:55:35,170 --> 00:55:37,045
before we got very far
into development where

1413
00:55:37,045 --> 00:55:39,290
we could pivot and make sure
that the idea of the two

1414
00:55:39,290 --> 00:55:42,330
guns doing the same
thing was conveyed.

1415
00:55:42,330 --> 00:55:44,016
So based on the
feedback we got, we

1416
00:55:44,016 --> 00:55:45,640
were able to give
that to the designers

1417
00:55:45,640 --> 00:55:47,800
and they continued
their development.

1418
00:55:47,800 --> 00:55:49,550
And then after a
little bit of time,

1419
00:55:49,550 --> 00:55:51,250
we were able to take
it into the lab.

1420
00:55:51,250 --> 00:55:53,190
So we started with a
Lucian that was designed

1421
00:55:53,190 --> 00:55:54,980
to be a speedy range assassin.

1422
00:55:54,980 --> 00:55:58,130
He had a double shot
and a utility kit.

1423
00:55:58,130 --> 00:55:59,880
And when I refer to
kit, just to be clear,

1424
00:55:59,880 --> 00:56:01,640
that means his set of abilities.

1425
00:56:01,640 --> 00:56:03,140
And characters in
League of Legends

1426
00:56:03,140 --> 00:56:05,830
generally have three abilities
and then an ultimate ability,

1427
00:56:05,830 --> 00:56:08,280
where the ultimate is like
the super big bad power

1428
00:56:08,280 --> 00:56:11,530
that you get after some
leveling of the character.

1429
00:56:11,530 --> 00:56:14,870
So I blew the reveal
here, but in the first lab

1430
00:56:14,870 --> 00:56:18,140
we found out that he didn't
really feel that sleek

1431
00:56:18,140 --> 00:56:18,810
and fast.

1432
00:56:18,810 --> 00:56:19,840
He felt stiff.

1433
00:56:19,840 --> 00:56:22,940
He planted and then fired
his double shot, which

1434
00:56:22,940 --> 00:56:25,220
made him feel like a turret.

1435
00:56:25,220 --> 00:56:26,990
And people did like
his visual theme.

1436
00:56:26,990 --> 00:56:28,781
They thought it was
compelling, but it just

1437
00:56:28,781 --> 00:56:30,790
didn't match his play style.

1438
00:56:30,790 --> 00:56:33,500
So we took this back to the
designers we said, hey guys.

1439
00:56:33,500 --> 00:56:34,550
Here's the videos.

1440
00:56:34,550 --> 00:56:35,587
Here's the notes.

1441
00:56:35,587 --> 00:56:37,170
We usually make a
highlight reel based

1442
00:56:37,170 --> 00:56:39,052
on what we see to
support our evidence so

1443
00:56:39,052 --> 00:56:41,260
that the designers can get
enough feedback from that,

1444
00:56:41,260 --> 00:56:43,060
as they are our
subject matter experts

1445
00:56:43,060 --> 00:56:45,924
and really know how to
interpret what they see.

1446
00:56:45,924 --> 00:56:46,590
And we said, OK.

1447
00:56:46,590 --> 00:56:47,450
Let's go back.

1448
00:56:47,450 --> 00:56:49,300
You guys take some
time, do some changes.

1449
00:56:49,300 --> 00:56:51,250
We'll take it in the lab again.

1450
00:56:51,250 --> 00:56:52,290
So we did lab study two.

1451
00:56:52,290 --> 00:56:54,780
And this time, we had a
different Lucian to bring in.

1452
00:56:54,780 --> 00:56:56,100
This time, he was more mobile.

1453
00:56:56,100 --> 00:56:58,060
He was basically a
range [INAUDIBLE],

1454
00:56:58,060 --> 00:57:01,740
running gun, double shot guy.

1455
00:57:01,740 --> 00:57:04,760
Well, we had a little bit of
a mixed bag of results here.

1456
00:57:04,760 --> 00:57:07,820
He did feel a little bit faster.

1457
00:57:07,820 --> 00:57:11,430
But some poses and
animations felt awkward.

1458
00:57:11,430 --> 00:57:14,611
And he was high speed
in the numbers-- which

1459
00:57:14,611 --> 00:57:16,360
I'll talk about in a
second-- but he still

1460
00:57:16,360 --> 00:57:18,580
felt slow for some reason.

1461
00:57:18,580 --> 00:57:21,550
But worst of all, we
got quotes like this one

1462
00:57:21,550 --> 00:57:23,120
about the ultimate,
which was "It

1463
00:57:23,120 --> 00:57:25,720
worked in the way I did
not want it to work,"

1464
00:57:25,720 --> 00:57:29,607
which is a pretty good sign
that something's wrong.

1465
00:57:29,607 --> 00:57:31,440
So we took all this
back to designers again.

1466
00:57:31,440 --> 00:57:33,100
We said, hey guys--
and the animators

1467
00:57:33,100 --> 00:57:35,349
and the artists in this case,
of course-- and we said,

1468
00:57:35,349 --> 00:57:36,040
he feels slow.

1469
00:57:36,040 --> 00:57:37,440
His ultimate doesn't
really have the usability

1470
00:57:37,440 --> 00:57:39,930
that people expect, but the
double shot theme is working,

1471
00:57:39,930 --> 00:57:41,513
so we're definitely
on the right track

1472
00:57:41,513 --> 00:57:44,440
and players are still very
excited about this champion.

1473
00:57:44,440 --> 00:57:46,760
So now we're in a
bind, and this is often

1474
00:57:46,760 --> 00:57:50,220
the case when you're working
with research from players

1475
00:57:50,220 --> 00:57:52,185
because they're not
always able to articulate

1476
00:57:52,185 --> 00:57:53,810
exactly what's wrong
because they don't

1477
00:57:53,810 --> 00:57:55,526
have the context that you do.

1478
00:57:55,526 --> 00:57:57,150
So this is why it's
so important for us

1479
00:57:57,150 --> 00:57:59,740
as researchers to pair with
our designers, our artists,

1480
00:57:59,740 --> 00:58:02,130
our producers, and
our players to be

1481
00:58:02,130 --> 00:58:03,840
able to get the
right kind of data

1482
00:58:03,840 --> 00:58:05,800
to make informed decisions.

1483
00:58:05,800 --> 00:58:07,870
So here was our problem.

1484
00:58:07,870 --> 00:58:10,240
The frame delay between
his basic attacks--

1485
00:58:10,240 --> 00:58:13,330
which is actually very,
very low in this last lab.

1486
00:58:13,330 --> 00:58:15,620
Usually that means
things feel fast.

1487
00:58:15,620 --> 00:58:18,920
It's a pretty standard way when
we're developing characters

1488
00:58:18,920 --> 00:58:21,280
to be like, OK, this
will feel faster.

1489
00:58:21,280 --> 00:58:24,520
He also had a very high
base movement speed,

1490
00:58:24,520 --> 00:58:27,650
which means that the time it
took him to run from point A

1491
00:58:27,650 --> 00:58:29,900
to point B was
pretty high relative

1492
00:58:29,900 --> 00:58:31,286
to the rest of our champions.

1493
00:58:31,286 --> 00:58:32,910
And if that weren't
enough, he actually

1494
00:58:32,910 --> 00:58:35,090
had a dash ability which
means that he could just

1495
00:58:35,090 --> 00:58:37,120
jump basically from
one point to another.

1496
00:58:37,120 --> 00:58:40,540
All of these things should
have made him feel fast.

1497
00:58:40,540 --> 00:58:42,660
But players were saying
he didn't feel fast,

1498
00:58:42,660 --> 00:58:45,840
so there was something wrong.

1499
00:58:45,840 --> 00:58:49,590
So our designers and our
artists and our producers

1500
00:58:49,590 --> 00:58:52,660
took all of this information
back, looked at the videos,

1501
00:58:52,660 --> 00:58:56,760
play tested the heck out of it,
and came to this conclusion.

1502
00:58:56,760 --> 00:58:59,410
They increased the
ability responsiveness

1503
00:58:59,410 --> 00:59:03,050
and they changed how animation
flowed from one state

1504
00:59:03,050 --> 00:59:04,030
to another.

1505
00:59:04,030 --> 00:59:05,727
So I'm not an
artist, but what this

1506
00:59:05,727 --> 00:59:07,310
means to me is that
they are basically

1507
00:59:07,310 --> 00:59:10,860
able to make the transitions
from animation a, animation

1508
00:59:10,860 --> 00:59:13,490
b, or ability a to
ability b, and make

1509
00:59:13,490 --> 00:59:15,450
those feel more natural.

1510
00:59:15,450 --> 00:59:17,020
Not like me.

1511
00:59:17,020 --> 00:59:22,510
And they also gave more
feedback to the double shot.

1512
00:59:22,510 --> 00:59:24,690
And again, this is where
subject matter expertise is

1513
00:59:24,690 --> 00:59:27,790
so important because our
animators, our visual artists,

1514
00:59:27,790 --> 00:59:30,300
our designers knew
this bag of tricks

1515
00:59:30,300 --> 00:59:32,780
to be able to apply to it.

1516
00:59:32,780 --> 00:59:34,640
And finally, they
were able to make

1517
00:59:34,640 --> 00:59:36,370
the ultimate ability clearer.

1518
00:59:36,370 --> 00:59:37,860
So they were able
to the telegraph

1519
00:59:37,860 --> 00:59:39,760
better what it was
actually supposed

1520
00:59:39,760 --> 00:59:44,120
to do so that players would
have more accurate expectations.

1521
00:59:44,120 --> 00:59:46,330
So we took it to lab
study number three.

1522
00:59:46,330 --> 00:59:48,815
We put it in and
finally, the game

1523
00:59:48,815 --> 00:59:50,910
play anesthetics
fit the concept,

1524
00:59:50,910 --> 00:59:52,810
and players were quite happy.

1525
00:59:52,810 --> 00:59:55,500
But not everything is
always happy ending,

1526
00:59:55,500 --> 00:59:56,990
everything is perfect.

1527
00:59:56,990 --> 01:00:00,080
There was still some confusion
about how the ultimate was

1528
01:00:00,080 --> 01:00:01,790
supposed to work.

1529
01:00:01,790 --> 01:00:03,950
We found that players
wanted more control of it.

1530
01:00:03,950 --> 01:00:05,700
Basically how the
ultimate currently works

1531
01:00:05,700 --> 01:00:09,140
is he locks his arms forward
and shoots in that direction.

1532
01:00:09,140 --> 01:00:10,839
So if you stray
left to right, he

1533
01:00:10,839 --> 01:00:12,630
continues shooting in
that direction which,

1534
01:00:12,630 --> 01:00:14,570
for some players,
was pretty confusing.

1535
01:00:14,570 --> 01:00:17,250
Now what they expected based
on some of our other champions,

1536
01:00:17,250 --> 01:00:20,249
and it wasn't able for them
to learn it in one session.

1537
01:00:20,249 --> 01:00:22,165
So one of the things
that we have to take away

1538
01:00:22,165 --> 01:00:25,590
from that is labs have some
external validity problems,

1539
01:00:25,590 --> 01:00:28,020
which basically means that
we're sitting in a lab.

1540
01:00:28,020 --> 01:00:29,470
There's not other
players around.

1541
01:00:29,470 --> 01:00:31,250
They're often not
playing on live.

1542
01:00:31,250 --> 01:00:33,500
There's a lot of things that
are different in our test

1543
01:00:33,500 --> 01:00:35,132
situations than real life.

1544
01:00:35,132 --> 01:00:36,590
Usually, you wouldn't
try to master

1545
01:00:36,590 --> 01:00:38,200
a champion in one session.

1546
01:00:38,200 --> 01:00:40,380
So this is one of those
cases where we put it out

1547
01:00:40,380 --> 01:00:43,430
to our designers and they
decided, and we supported,

1548
01:00:43,430 --> 01:00:47,822
that he was going to be a high
skill ceiling champion, which

1549
01:00:47,822 --> 01:00:49,780
meant that players were
going to need some time

1550
01:00:49,780 --> 01:00:51,060
to learn how to play him.

1551
01:00:51,060 --> 01:00:53,590
And that just because they
didn't learn him in one session

1552
01:00:53,590 --> 01:00:55,714
didn't mean that they
wouldn't learn him over time.

1553
01:00:59,050 --> 01:01:03,560
So this is what Lucian looked
like when he did go live.

1554
01:01:03,560 --> 01:01:05,067
And it helps if
I play the video.

1555
01:01:08,710 --> 01:01:10,120
[VIDEO PLAYBACK]

1556
01:01:10,120 --> 01:01:11,060
[WARFARE SOUNDS]

1557
01:01:11,060 --> 01:01:14,039
GENEVIEVE CONLEY: So
he's in the top there.

1558
01:01:14,039 --> 01:01:15,971
Pew, pew.

1559
01:01:15,971 --> 01:01:19,352
[WARFARE SOUNDS]

1560
01:01:22,720 --> 01:01:24,720
GENEVIEVE CONLEY: So you
can see his dash there.

1561
01:01:28,504 --> 01:01:30,099
[END PLAYBACK]

1562
01:01:30,099 --> 01:01:32,390
GENEVIEVE CONLEY: So yeah,
sometimes a little confusing

1563
01:01:32,390 --> 01:01:33,806
if you don't play
the game to have

1564
01:01:33,806 --> 01:01:36,480
to watch 10 champions all in
one little team fight like that.

1565
01:01:36,480 --> 01:01:38,350
But ultimately, the
important part's there.

1566
01:01:38,350 --> 01:01:40,914
He's using his ultimate
and he dashed at the end.

1567
01:01:40,914 --> 01:01:42,580
Ultimately, ended up
actually delivering

1568
01:01:42,580 --> 01:01:44,400
on player expectations
for this champion.

1569
01:01:44,400 --> 01:01:45,577
[VIDEO PLAYBACK]

1570
01:01:45,577 --> 01:01:46,160
[END PLAYBACK]

1571
01:01:46,160 --> 01:01:49,210
We don't need to
watch that again.

1572
01:01:49,210 --> 01:01:50,440
So what did we learn?

1573
01:01:50,440 --> 01:01:52,220
Well, we knew going
into it that Lucian

1574
01:01:52,220 --> 01:01:54,304
was going to be a difficult
champion to get right.

1575
01:01:54,304 --> 01:01:55,720
And we also learned
that, you know

1576
01:01:55,720 --> 01:01:58,530
what, it's OK to take a champion
back to the lab multiple times

1577
01:01:58,530 --> 01:02:01,837
if we really feel that
it's worth the effort.

1578
01:02:01,837 --> 01:02:03,420
We knew that his
theme was compelling,

1579
01:02:03,420 --> 01:02:05,700
but there was something
that wasn't just matching up

1580
01:02:05,700 --> 01:02:08,712
between his mechanics, his
visuals, and his theme.

1581
01:02:08,712 --> 01:02:11,170
And this is one of those things
is that everything on paper

1582
01:02:11,170 --> 01:02:14,360
can be totally 100%
right, but if players

1583
01:02:14,360 --> 01:02:16,547
don't agree with it or
players aren't feeling

1584
01:02:16,547 --> 01:02:18,130
those goals that
you're trying to hit,

1585
01:02:18,130 --> 01:02:19,830
there's going to be a problem.

1586
01:02:19,830 --> 01:02:22,270
So we're able to collaborate
with our subject matter

1587
01:02:22,270 --> 01:02:24,630
experts, with our
players, in order

1588
01:02:24,630 --> 01:02:27,230
to get the experience right.

1589
01:02:27,230 --> 01:02:30,470
And I mean, I'll say it again.

1590
01:02:30,470 --> 01:02:32,620
Just because the mechanics
stacked up correctly

1591
01:02:32,620 --> 01:02:34,330
does not mean that
it necessarily will

1592
01:02:34,330 --> 01:02:37,020
in the feeling, which is why
you play test it in house,

1593
01:02:37,020 --> 01:02:39,280
why you play test it with
other people at your studio

1594
01:02:39,280 --> 01:02:41,630
or at your school, and
ideally, why you test it

1595
01:02:41,630 --> 01:02:44,650
with the people who are actually
going to play your game.

1596
01:02:44,650 --> 01:02:46,780
And then we also learned
that you can sometimes

1597
01:02:46,780 --> 01:02:48,280
have these trade-offs.

1598
01:02:48,280 --> 01:02:51,060
It goes back to what
you hear from players.

1599
01:02:51,060 --> 01:02:52,950
Sometimes they're going
to make suggestions

1600
01:02:52,950 --> 01:02:55,140
and you have to look for
the note behind the note.

1601
01:02:55,140 --> 01:02:56,904
It's like, what is
actually the problem?

1602
01:02:56,904 --> 01:02:58,570
You guys are the
subject matter experts.

1603
01:02:58,570 --> 01:03:00,819
You're going to be able to
look at what they're saying

1604
01:03:00,819 --> 01:03:03,820
and never take it as,
from players, a criticism.

1605
01:03:03,820 --> 01:03:06,630
Always think about it as if
they said something wrong,

1606
01:03:06,630 --> 01:03:09,170
there's something that has
to be causing the problem.

1607
01:03:09,170 --> 01:03:11,070
What is it underneath
their note that's

1608
01:03:11,070 --> 01:03:12,660
actually causing that problem?

1609
01:03:12,660 --> 01:03:13,880
Let's look for that thing.

1610
01:03:16,630 --> 01:03:18,850
And we're still
iterating after launch.

1611
01:03:18,850 --> 01:03:21,590
That's one thing that's typical
with our champions is that yes,

1612
01:03:21,590 --> 01:03:22,840
we do play tests with players.

1613
01:03:22,840 --> 01:03:25,160
Yes, we do play tests in house.

1614
01:03:25,160 --> 01:03:27,300
But once we launch
the champion, there

1615
01:03:27,300 --> 01:03:29,600
are thousands and
thousands of play tests

1616
01:03:29,600 --> 01:03:31,699
with real players on live.

1617
01:03:31,699 --> 01:03:33,990
And that's when we learn
about things that happen maybe

1618
01:03:33,990 --> 01:03:37,080
at scale or under very
specific situations,

1619
01:03:37,080 --> 01:03:38,900
or for things that we
didn't account for,

1620
01:03:38,900 --> 01:03:41,330
for these external
validity concerns.

1621
01:03:41,330 --> 01:03:43,020
So we keep iterating.

1622
01:03:43,020 --> 01:03:44,640
And in the case of
Lucian, we actually

1623
01:03:44,640 --> 01:03:47,360
continue to collect
metrics and user sentiment

1624
01:03:47,360 --> 01:03:49,570
through things like surveys
and through game data

1625
01:03:49,570 --> 01:03:51,309
to understand how he was doing.

1626
01:03:51,309 --> 01:03:52,850
And based on this,
we're able to make

1627
01:03:52,850 --> 01:03:54,990
more iterations, fine
tune, and hopefully

1628
01:03:54,990 --> 01:03:57,300
make him a little
bit more accessible.

1629
01:03:57,300 --> 01:03:59,790
And today, Lucian is well
regarded, widely played.

1630
01:03:59,790 --> 01:04:01,520
And if you watch the
professional matches

1631
01:04:01,520 --> 01:04:04,364
in the world championship,
he's use quite frequently.

1632
01:04:04,364 --> 01:04:06,280
All right, you guys, I
hope I didn't bore you.

1633
01:04:06,280 --> 01:04:07,491
My name is Genevieve, again.

1634
01:04:07,491 --> 01:04:08,990
If you guys ever
have any questions,

1635
01:04:08,990 --> 01:04:12,510
you can feel free to reach out
to me at gconley@riotgames.com.

1636
01:04:12,510 --> 01:04:15,190
My Summoner name in game, if
you play, is RiotCapernoited.

1637
01:04:15,190 --> 01:04:16,510
And you can also find
me on Twitter there.

1638
01:04:16,510 --> 01:04:18,051
I'm always happy to
answer questions.

1639
01:04:18,051 --> 01:04:19,284
Thank you guys.

1640
01:04:19,284 --> 01:04:23,140
[APPLAUSE]

1641
01:04:27,960 --> 01:04:30,870
AUDIENCE: We get
a 10 minute break.

1642
01:04:30,870 --> 01:04:33,840
PROFESSOR: So 10 minute break.

1643
01:04:33,840 --> 01:04:38,110
Come back at 2:20 and be ready
to start working on your focus

1644
01:04:38,110 --> 01:04:40,320
test stuff.

1645
01:04:40,320 --> 01:04:43,790
OK, welcome back to class.

1646
01:04:46,590 --> 01:04:49,180
Now that you've had a chance
to be talked at a whole lot,

1647
01:04:49,180 --> 01:04:54,090
we're going to go ahead
and ask you to actually do.

1648
01:04:54,090 --> 01:04:57,660
The plan is to give you about 10
minutes to meet in your groups,

1649
01:04:57,660 --> 01:04:59,940
plan your test.

1650
01:04:59,940 --> 01:05:01,850
I actually strongly
recommend you just grab

1651
01:05:01,850 --> 01:05:05,250
a copy of the focus test,
either in the Word version

1652
01:05:05,250 --> 01:05:06,800
or the RTF version
of off Stellar

1653
01:05:06,800 --> 01:05:08,810
so you've got that as a guide.

1654
01:05:08,810 --> 01:05:12,712
Sit down, plan your test and
how you're going to run it.

1655
01:05:12,712 --> 01:05:14,170
We'll give you 10
minutes for that.

1656
01:05:14,170 --> 01:05:16,710
Then we'll have about
45 minutes for people

1657
01:05:16,710 --> 01:05:18,010
to circulate and test.

1658
01:05:18,010 --> 01:05:20,051
And one of the things you
should also be planning

1659
01:05:20,051 --> 01:05:21,650
is when your team
members will be

1660
01:05:21,650 --> 01:05:24,400
out testing other people's
games and when your team members

1661
01:05:24,400 --> 01:05:26,850
will be observing.

1662
01:05:26,850 --> 01:05:29,860
Just given the number of people
in class, the number of games

1663
01:05:29,860 --> 01:05:32,990
we have, and the number
of people on the team,

1664
01:05:32,990 --> 01:05:35,560
chances are good that you aren't
going to be either observing

1665
01:05:35,560 --> 01:05:37,560
or testing all the time.

1666
01:05:37,560 --> 01:05:40,410
You're probably going
to have to wait.

1667
01:05:40,410 --> 01:05:42,380
But go ahead and
circulate, take the time

1668
01:05:42,380 --> 01:05:45,190
to watch other teams
doing focus testing.

1669
01:05:45,190 --> 01:05:48,200
Try not to watch
a game in progress

1670
01:05:48,200 --> 01:05:51,140
that you're going to focus test.

1671
01:05:51,140 --> 01:05:53,230
If you realize that
you're just sitting

1672
01:05:53,230 --> 01:05:54,980
around waiting for a
chance to focus test,

1673
01:05:54,980 --> 01:05:57,630
let the team know that
you're in line next.

1674
01:05:57,630 --> 01:06:00,540
I'd rather you pulled
out a book and read it

1675
01:06:00,540 --> 01:06:02,830
than watched someone
else play a game

1676
01:06:02,830 --> 01:06:04,120
you're about to focus test.

1677
01:06:04,120 --> 01:06:06,328
It will be more useful for
the poor people who you're

1678
01:06:06,328 --> 01:06:08,430
testing for if you
haven't already seen

1679
01:06:08,430 --> 01:06:10,521
somebody do all those things.

1680
01:06:10,521 --> 01:06:12,020
That doesn't mean
hide in the corner

1681
01:06:12,020 --> 01:06:14,930
and read a book because
there's too many people

1682
01:06:14,930 --> 01:06:16,580
wandering around testing games.

1683
01:06:16,580 --> 01:06:20,250
We do expect everybody to
participate both sides.

1684
01:06:20,250 --> 01:06:22,180
Observe and test.

1685
01:06:22,180 --> 01:06:25,020
So try to balance
that out a bit.

1686
01:06:25,020 --> 01:06:27,250
We'll circulate and
keep an eye on things.

1687
01:06:27,250 --> 01:06:29,690
And if it looks like at
some point nobody's actually

1688
01:06:29,690 --> 01:06:32,550
testing, you're all sitting
around twiddling your thumbs,

1689
01:06:32,550 --> 01:06:33,390
we'll end it early.

1690
01:06:33,390 --> 01:06:36,400
I don't know how long
this testing session will

1691
01:06:36,400 --> 01:06:40,580
take because we haven't had
such large teams on this project

1692
01:06:40,580 --> 01:06:41,080
before.

1693
01:06:41,080 --> 01:06:44,330
So it's a little harder for me
to know what the amount of time

1694
01:06:44,330 --> 01:06:46,570
is.

1695
01:06:46,570 --> 01:06:48,310
After that, we'll give
you some more time

1696
01:06:48,310 --> 01:06:51,280
to talk in your own
teams and come up

1697
01:06:51,280 --> 01:06:54,084
with the what are the results of
your test to analyze your data.

1698
01:06:54,084 --> 01:06:55,750
And then we'll ask
somebody to come down

1699
01:06:55,750 --> 01:06:58,440
and for each team give us the
summary of your focus test

1700
01:06:58,440 --> 01:06:59,456
report.

1701
01:06:59,456 --> 01:07:00,830
How did your focus
test turn out?

1702
01:07:00,830 --> 01:07:01,622
What did you learn?

1703
01:07:01,622 --> 01:07:03,913
Do you have a plan for what
you're going to do with it,

1704
01:07:03,913 --> 01:07:06,510
or are you still figuring
out how you'll deal with it?

1705
01:07:06,510 --> 01:07:08,200
Those are all
reasonable things given

1706
01:07:08,200 --> 01:07:10,640
you only had 10 minutes
to look at your data.

1707
01:07:10,640 --> 01:07:11,140
Thank you.

1708
01:07:11,140 --> 01:07:13,700
It is 2:23.

1709
01:07:13,700 --> 01:07:18,631
At 2:33, we'll go ahead
and start focus testing.

1710
01:07:18,631 --> 01:07:20,380
Actually, I'm going
to give you 15 minutes

1711
01:07:20,380 --> 01:07:22,505
to get set up, because I
just realized you probably

1712
01:07:22,505 --> 01:07:25,652
need 10 minutes to talk and
then probably five minutes

1713
01:07:25,652 --> 01:07:27,610
to make sure you have a
machine up and running.

1714
01:07:27,610 --> 01:07:30,920
So I apologize, it's going
to be 15 minutes to plan

1715
01:07:30,920 --> 01:07:33,170
and prepare for your test.

1716
01:07:33,170 --> 01:07:37,258
So that's 2:38 at this point.

1717
01:07:37,258 --> 01:07:41,250
[CHATTER]

1718
01:07:52,126 --> 01:07:52,750
AUDIENCE: Yeah.

1719
01:07:52,750 --> 01:07:53,930
I think he just wants to--

1720
01:07:53,930 --> 01:07:54,610
PROFESSOR: OK.

1721
01:07:54,610 --> 01:07:58,080
You should be ready
to go now, I hope.

1722
01:07:58,080 --> 01:08:00,650
And actively go ahead.

1723
01:08:00,650 --> 01:08:03,070
You should have a
computer ideally set up

1724
01:08:03,070 --> 01:08:05,170
to run your game.

1725
01:08:05,170 --> 01:08:08,200
Who is actually ready
to run a focus test?

1726
01:08:08,200 --> 01:08:10,740
PHILIP TAN: Everyone who
is manning a demo computer,

1727
01:08:10,740 --> 01:08:15,130
a computer ready to run
the game, put your hand up.

1728
01:08:15,130 --> 01:08:16,910
All stand up so
that we can see you.

1729
01:08:16,910 --> 01:08:17,445
PROFESSOR: Yeah, go
ahead and stay up so

1730
01:08:17,445 --> 01:08:18,980
we can see where you are.

1731
01:08:18,980 --> 01:08:20,480
PHILIP TAN: I'm not
seeing any hands

1732
01:08:20,480 --> 01:08:22,230
from these teams over here.

1733
01:08:22,230 --> 01:08:22,729
OK.

1734
01:08:22,729 --> 01:08:23,000
All right.

1735
01:08:23,000 --> 01:08:23,240
All right.

1736
01:08:23,240 --> 01:08:23,740
All right.

1737
01:08:23,740 --> 01:08:26,970
As I said that, someone
is standing up back there.

1738
01:08:26,970 --> 01:08:30,997
How about the Sparkly
Redemption team?

1739
01:08:30,997 --> 01:08:31,580
OK, all right.

1740
01:08:31,580 --> 01:08:32,870
There's someone
there to run a game.

1741
01:08:32,870 --> 01:08:33,470
All right.

1742
01:08:33,470 --> 01:08:35,290
PROFESSOR: So close
all the laptops

1743
01:08:35,290 --> 01:08:38,990
that are not actually running
the game would be my advice,

1744
01:08:38,990 --> 01:08:43,160
and then let's go
ahead and get started.

1745
01:08:43,160 --> 01:08:44,160
PHILIP TAN: All right.

1746
01:08:44,160 --> 01:08:45,770
Start playing each
other's games.

1747
01:08:45,770 --> 01:08:46,270
Go.

1748
01:08:46,270 --> 01:08:47,758
PROFESSOR: Yep.

1749
01:08:47,758 --> 01:08:52,222
[CHATTER]

1750
01:09:02,280 --> 01:09:04,696
PROFESSOR: Next time, we'll
ask them to have two or three.

1751
01:09:04,696 --> 01:09:06,640
PHILIP TAN: Yeah.

1752
01:09:06,640 --> 01:09:09,149
PROFESSOR: As part
of the Stellar thing.

1753
01:09:09,149 --> 01:09:10,428
He was talking about the--

1754
01:09:10,428 --> 01:09:14,332
AUDIENCE: I got
the FOA for free.

1755
01:09:14,332 --> 01:09:14,915
PROFESSOR: OK.

1756
01:09:19,330 --> 01:09:22,170
It's OK if you don't
have a plan of action

1757
01:09:22,170 --> 01:09:23,600
as you're looking
at your results

1758
01:09:23,600 --> 01:09:27,660
as long as you've gotten ideas
to what you're seeing out

1759
01:09:27,660 --> 01:09:29,300
of your data.

1760
01:09:29,300 --> 01:09:32,424
It's hard to look at your
data, get it evaluated,

1761
01:09:32,424 --> 01:09:33,840
talk about it, and
figure out what

1762
01:09:33,840 --> 01:09:35,569
you're going to do about
it all in 10 minutes.

1763
01:09:35,569 --> 01:09:38,000
We are giving you a short
amount of time because we're

1764
01:09:38,000 --> 01:09:40,229
running out of time.

1765
01:09:40,229 --> 01:09:42,464
That said, let's get started
with Sparkly Redemption.

1766
01:09:45,179 --> 01:09:47,629
[APPLAUSE]

1767
01:09:47,629 --> 01:09:49,589
AUDIENCE: So we
had a small hiccup

1768
01:09:49,589 --> 01:09:52,845
when we started where we didn't
have-- part of our core game

1769
01:09:52,845 --> 01:09:55,469
is collecting sparklies, and we
didn't have a version with that

1770
01:09:55,469 --> 01:09:56,800
when we started play testing.

1771
01:09:56,800 --> 01:09:59,050
So we actually got to test
out some different movement

1772
01:09:59,050 --> 01:10:00,425
controls that we
were trying out,

1773
01:10:00,425 --> 01:10:03,390
which actually turned out to
be good because the controls

1774
01:10:03,390 --> 01:10:05,786
that we liked, which was
right clicking with the mouse

1775
01:10:05,786 --> 01:10:07,871
and then left clicking
to shoot, a lot of people

1776
01:10:07,871 --> 01:10:09,463
had a lot of trouble with that.

1777
01:10:09,463 --> 01:10:12,925
So it was a good thing that
we messed up and then got

1778
01:10:12,925 --> 01:10:14,590
some information from that.

1779
01:10:14,590 --> 01:10:17,650
Then we got a version
with sparklies working,

1780
01:10:17,650 --> 01:10:20,660
and there was a big bug where
if you killed all the monsters,

1781
01:10:20,660 --> 01:10:21,990
they never respond.

1782
01:10:21,990 --> 01:10:24,972
So that bug has now been
fixed in the last 10 minutes,

1783
01:10:24,972 --> 01:10:26,463
so that was good.

1784
01:10:26,463 --> 01:10:30,594
Other than we, we got just
random things like people

1785
01:10:30,594 --> 01:10:32,260
staying in the corner
and just shooting,

1786
01:10:32,260 --> 01:10:34,970
as opposed to just moving around
the map like we want them to.

1787
01:10:34,970 --> 01:10:38,357
So we have a few problems that
we have to figure out that.

1788
01:10:38,357 --> 01:10:41,351
Right now, we don't
know the answers.

1789
01:10:41,351 --> 01:10:44,844
[APPLAUSE]

1790
01:10:50,333 --> 01:10:51,097
AUDIENCE: Hi.

1791
01:10:51,097 --> 01:10:52,180
I'm with Lost Underground.

1792
01:10:52,180 --> 01:10:54,385
As a refresher on
our game, the idea

1793
01:10:54,385 --> 01:10:57,472
is that you are lost underground
in the darkness trying

1794
01:10:57,472 --> 01:10:58,602
to find an exit.

1795
01:10:58,602 --> 01:11:02,665
And so as part of our
game, we have this sphere

1796
01:11:02,665 --> 01:11:04,418
of visibility around you.

1797
01:11:04,418 --> 01:11:07,490
And as we play tested, we
got a lot of good feedback

1798
01:11:07,490 --> 01:11:09,935
on what that visibility
means to people

1799
01:11:09,935 --> 01:11:12,883
and how we can
actually use it to make

1800
01:11:12,883 --> 01:11:15,751
our mechanics and our
[INAUDIBLE] design

1801
01:11:15,751 --> 01:11:16,950
more compelling.

1802
01:11:16,950 --> 01:11:19,614
Another big issue we
ran into with people

1803
01:11:19,614 --> 01:11:24,035
play testing was
our collision boxes

1804
01:11:24,035 --> 01:11:25,430
are basically pixel perfect.

1805
01:11:25,430 --> 01:11:27,180
And so people would
get caught trying

1806
01:11:27,180 --> 01:11:32,370
to go through the map, which
was really helpful for us,

1807
01:11:32,370 --> 01:11:35,017
because although we already
knew that this problem existed,

1808
01:11:35,017 --> 01:11:37,350
we actually learned a lot
about how people were actually

1809
01:11:37,350 --> 01:11:38,844
trying to make these decisions.

1810
01:11:38,844 --> 01:11:40,587
And we learned that
instead of using

1811
01:11:40,587 --> 01:11:43,630
the idea of the entire
[INAUDIBLE], a lot of players

1812
01:11:43,630 --> 01:11:46,330
were actually looking at
where the feet were planted

1813
01:11:46,330 --> 01:11:47,776
and trying to move
based on that.

1814
01:11:47,776 --> 01:11:50,186
So when we go and
fix it, we'll make

1815
01:11:50,186 --> 01:11:53,240
sure to put the collision box
around where the feet are.

1816
01:11:53,240 --> 01:11:56,335
We also got a lot of good
feedback on the items

1817
01:11:56,335 --> 01:11:58,584
that we have in our game and
where people are actually

1818
01:11:58,584 --> 01:12:01,310
focusing their attention.

1819
01:12:01,310 --> 01:12:04,677
[APPLAUSE]

1820
01:12:08,044 --> 01:12:09,390
AUDIENCE: [INAUDIBLE].

1821
01:12:09,390 --> 01:12:10,602
PROFESSOR: Oh, Plunder Winds.

1822
01:12:10,602 --> 01:12:11,102
Yeah.

1823
01:12:17,130 --> 01:12:22,322
AUDIENCE: So our game
involves moving around a grid

1824
01:12:22,322 --> 01:12:24,210
and running into
random encounters.

1825
01:12:27,167 --> 01:12:28,875
The favorability of
the random encounters

1826
01:12:28,875 --> 01:12:32,230
are indicated to the player
by numbers and colors

1827
01:12:32,230 --> 01:12:33,770
on the board.

1828
01:12:33,770 --> 01:12:36,675
So one of the biggest things
that we got from testing

1829
01:12:36,675 --> 01:12:38,100
was that people
didn't understand

1830
01:12:38,100 --> 01:12:39,808
the role of the numbers
and colors played

1831
01:12:39,808 --> 01:12:42,486
or how they were related
or if they were related.

1832
01:12:42,486 --> 01:12:49,025
And they began by moving around
randomly and just waiting

1833
01:12:49,025 --> 01:12:51,138
for things to happen to them.

1834
01:12:51,138 --> 01:12:52,596
There's also a
mechanic in our game

1835
01:12:52,596 --> 01:12:57,347
that involves moving with the
wind and against the wind.

1836
01:12:57,347 --> 01:12:59,430
And so moving with the
wind doesn't count stamina,

1837
01:12:59,430 --> 01:13:02,604
but moving against
the wind does.

1838
01:13:02,604 --> 01:13:04,666
Or moving sideways
of the wind does.

1839
01:13:04,666 --> 01:13:06,332
Moving against the
wind is not possible.

1840
01:13:06,332 --> 01:13:09,110
People had trouble
figuring that out as well.

1841
01:13:09,110 --> 01:13:11,735
So one of the biggest things
we're going to try and do

1842
01:13:11,735 --> 01:13:15,550
is more visual
feedback to the player,

1843
01:13:15,550 --> 01:13:18,352
help them figure
things out as it goes.

1844
01:13:18,352 --> 01:13:19,810
Also, people felt
like there should

1845
01:13:19,810 --> 01:13:23,710
be more long-term consequences
instead of just planning

1846
01:13:23,710 --> 01:13:25,323
for the next move or two.

1847
01:13:25,323 --> 01:13:27,788
And so we're going to see
how we can integrate that

1848
01:13:27,788 --> 01:13:30,253
into our game.

1849
01:13:30,253 --> 01:13:33,704
[APPLAUSE]

1850
01:13:39,620 --> 01:13:40,606
AUDIENCE: Hi.

1851
01:13:40,606 --> 01:13:41,592
Blind Aliens.

1852
01:13:41,592 --> 01:13:43,564
So we got a lot
of great feedback

1853
01:13:43,564 --> 01:13:45,740
from the use play testing.

1854
01:13:45,740 --> 01:13:47,440
So one of the main
things we realized

1855
01:13:47,440 --> 01:13:52,148
is we needed more clear goals
and more long-term goals.

1856
01:13:52,148 --> 01:13:54,549
So for example, our gun
automatically reloads,

1857
01:13:54,549 --> 01:13:56,298
and that was a little
confusing for people

1858
01:13:56,298 --> 01:13:58,960
because they didn't really
see that it was reloading.

1859
01:13:58,960 --> 01:14:00,638
But instead of doing
something like that

1860
01:14:00,638 --> 01:14:02,198
where the gun
automatically reloads,

1861
01:14:02,198 --> 01:14:04,772
we're thinking of adding
bullets around that you

1862
01:14:04,772 --> 01:14:08,490
have to go and collect so it's
more short-term goals and also

1863
01:14:08,490 --> 01:14:09,316
long-term strategy.

1864
01:14:09,316 --> 01:14:12,274
How do you save your bullets
up, when do you use them all.

1865
01:14:12,274 --> 01:14:14,037
Another thing we noticed
was that our game

1866
01:14:14,037 --> 01:14:15,620
was way too hard at
first, and way too

1867
01:14:15,620 --> 01:14:17,069
easy after five minutes.

1868
01:14:17,069 --> 01:14:20,450
So we decided we need to focus
on ramping up the difficulty.

1869
01:14:20,450 --> 01:14:22,623
Instead of starting with
three aliens attacking you,

1870
01:14:22,623 --> 01:14:23,706
maybe only one.

1871
01:14:23,706 --> 01:14:26,080
And then maybe after five
minutes, instead of only having

1872
01:14:26,080 --> 01:14:28,590
three aliens, maybe 10.

1873
01:14:28,590 --> 01:14:32,130
[INAUDIBLE] difficult.

1874
01:14:32,130 --> 01:14:37,486
So another thing we
focused on was our AI.

1875
01:14:37,486 --> 01:14:38,325
Not AI.

1876
01:14:38,325 --> 01:14:39,400
User interface.

1877
01:14:39,400 --> 01:14:42,220
So I talked about how there
was a reloading thing,

1878
01:14:42,220 --> 01:14:44,255
it was flashing in the
corner, but no one saw it

1879
01:14:44,255 --> 01:14:46,250
so no one knew that
their gun was reloading.

1880
01:14:46,250 --> 01:14:48,400
They thought it
just wasn't working

1881
01:14:48,400 --> 01:14:50,677
and they were frustrated
that they were clicking

1882
01:14:50,677 --> 01:14:51,760
and nothing was happening.

1883
01:14:51,760 --> 01:14:53,790
So we need to make
that more visual.

1884
01:14:53,790 --> 01:14:56,245
Along with our aliens,
when they're half-dead

1885
01:14:56,245 --> 01:14:57,610
or they're still attacking you.

1886
01:14:57,610 --> 01:14:58,860
People thought they were dead.

1887
01:14:58,860 --> 01:15:01,580
They'd run into them and
they'd die and be frustrated.

1888
01:15:01,580 --> 01:15:04,940
So instead of maybe
turning them darker,

1889
01:15:04,940 --> 01:15:07,830
maybe turn them red or something
like that to make that clearer.

1890
01:15:07,830 --> 01:15:09,889
So thank you guys
for all the feedback.

1891
01:15:09,889 --> 01:15:13,200
[APPLAUSE]

1892
01:15:14,237 --> 01:15:15,070
PROFESSOR: Modudice.

1893
01:15:21,171 --> 01:15:23,130
AUDIENCE: Hi, we're Modudice.

1894
01:15:23,130 --> 01:15:26,105
So a lot of the feedback we
got was really based on our UI.

1895
01:15:26,105 --> 01:15:32,070
And it's because we're modeling
a 3-D die using 2-D graphics.

1896
01:15:32,070 --> 01:15:34,447
So a lot of people
have different opinions

1897
01:15:34,447 --> 01:15:36,030
about how that should
look, and no one

1898
01:15:36,030 --> 01:15:38,477
thought that the way
it does look is OK.

1899
01:15:38,477 --> 01:15:39,760
[INAUDIBLE]

1900
01:15:39,760 --> 01:15:42,913
And so we really got
a lot of opinions.

1901
01:15:42,913 --> 01:15:44,204
Maybe you could do it this way.

1902
01:15:44,204 --> 01:15:45,495
Maybe you could do it this way.

1903
01:15:45,495 --> 01:15:47,876
And that really
helps, because now we

1904
01:15:47,876 --> 01:15:50,230
can really build up a discussion
of how it should look.

1905
01:15:50,230 --> 01:15:52,820
How we can make this mental
model in people's heads

1906
01:15:52,820 --> 01:15:54,750
match what they're seeing
on the screen, which

1907
01:15:54,750 --> 01:15:57,180
is the real hard challenge
that we're facing.

1908
01:15:57,180 --> 01:15:59,610
So thanks for the feedback.

1909
01:15:59,610 --> 01:16:03,500
[APPLAUSE]

1910
01:16:03,500 --> 01:16:05,850
PROFESSOR: Beaver Evolution.

1911
01:16:05,850 --> 01:16:08,350
PHILIP TAN: I told them it
was going to be the problem,

1912
01:16:08,350 --> 01:16:10,850
someone needs to go up.

1913
01:16:10,850 --> 01:16:12,850
AUDIENCE: So through
play testing,

1914
01:16:12,850 --> 01:16:16,580
we really realized that we
weren't conveying to the user

1915
01:16:16,580 --> 01:16:19,424
exactly why all the
instructions worked clearly

1916
01:16:19,424 --> 01:16:21,492
and what was happening
at each turn.

1917
01:16:21,492 --> 01:16:25,160
But otherwise, the game
mechanics worked pretty well.

1918
01:16:25,160 --> 01:16:27,640
So in Beaver Evolution,
you can choose

1919
01:16:27,640 --> 01:16:29,282
whether to build,
populate, or evolve,

1920
01:16:29,282 --> 01:16:31,247
but it wasn't
clear to the player

1921
01:16:31,247 --> 01:16:32,955
exactly how many
beavers you can populate

1922
01:16:32,955 --> 01:16:34,730
or how many dams
you could build,

1923
01:16:34,730 --> 01:16:37,720
or what natural
disasters could happen.

1924
01:16:37,720 --> 01:16:41,526
And then between each turn, when
a natural disaster happened,

1925
01:16:41,526 --> 01:16:43,020
there wasn't that
much engagement

1926
01:16:43,020 --> 01:16:45,880
between the player and
the game being like,

1927
01:16:45,880 --> 01:16:48,302
oh, this drought
happened and then

1928
01:16:48,302 --> 01:16:50,470
this is a consequence of that.

1929
01:16:50,470 --> 01:16:53,970
So I think going forward,
we're going to really try

1930
01:16:53,970 --> 01:16:57,399
to make it easier for players
to understand what happened

1931
01:16:57,399 --> 01:16:59,864
during that turn, and then
what their choices are

1932
01:16:59,864 --> 01:17:01,343
in terms of picking it.

1933
01:17:01,343 --> 01:17:04,794
So a lot of it is clarifying
what the instructions are,

1934
01:17:04,794 --> 01:17:07,752
what the game mechanics are.

1935
01:17:07,752 --> 01:17:11,203
[APPLAUSE]

1936
01:17:12,304 --> 01:17:13,720
PROFESSOR: And
finally, Comcastic.

1937
01:17:19,674 --> 01:17:21,132
AUDIENCE: Well,
the two main pieces

1938
01:17:21,132 --> 01:17:22,632
of feedback that
we got from players

1939
01:17:22,632 --> 01:17:24,940
were about the UI of the game.

1940
01:17:24,940 --> 01:17:27,450
And we had some issues with
that because our game has

1941
01:17:27,450 --> 01:17:30,354
a lot of complex UI
requirements and we

1942
01:17:30,354 --> 01:17:33,711
didn't have time to implement
all of those prior to this test

1943
01:17:33,711 --> 01:17:34,211
session.

1944
01:17:34,211 --> 01:17:36,760
But we did get a lot of feedback
about UI improvements that

1945
01:17:36,760 --> 01:17:39,445
would make it easier
for players to see what

1946
01:17:39,445 --> 01:17:40,840
was going on with the game.

1947
01:17:40,840 --> 01:17:42,381
For example,
highlighting houses that

1948
01:17:42,381 --> 01:17:44,570
are currently being served
by a [? source ?] center.

1949
01:17:44,570 --> 01:17:46,070
The other main
complaint that we got

1950
01:17:46,070 --> 01:17:48,597
was that the game is
nebulous right now.

1951
01:17:48,597 --> 01:17:50,096
There's not a whole
lot of challenge

1952
01:17:50,096 --> 01:17:51,203
or clearly defined goal.

1953
01:17:51,203 --> 01:17:52,676
And so one thing
that we're really

1954
01:17:52,676 --> 01:17:56,390
going to look at
going forward is

1955
01:17:56,390 --> 01:17:59,470
tightening the focus of the
game, addling more tension,

1956
01:17:59,470 --> 01:18:02,026
and making the goal of
the game more concrete.

1957
01:18:02,026 --> 01:18:06,328
And since it doesn't fit very
well into an established genre,

1958
01:18:06,328 --> 01:18:08,847
we had some more
complex design decisions

1959
01:18:08,847 --> 01:18:11,556
to figure out how to make
this a fun and engaging game.

1960
01:18:11,556 --> 01:18:13,055
And so that's pretty
much what we're

1961
01:18:13,055 --> 01:18:15,035
going to be focusing
on going forward,

1962
01:18:15,035 --> 01:18:18,005
would be the UI and adding
more challenge and tension

1963
01:18:18,005 --> 01:18:20,975
to the game.

1964
01:18:20,975 --> 01:18:22,460
[APPLAUSE]

1965
01:18:22,460 --> 01:18:23,945
PHILIP TAN: I have a few words.

1966
01:18:23,945 --> 01:18:25,879
Do you have anything to say?

1967
01:18:25,879 --> 01:18:26,920
PROFESSOR: You go for it.

1968
01:18:26,920 --> 01:18:27,545
PHILIP TAN: OK.

1969
01:18:27,545 --> 01:18:29,110
So I just want to
remind everybody

1970
01:18:29,110 --> 01:18:33,701
that you started on this
project less than 10 days ago.

1971
01:18:33,701 --> 01:18:34,200
OK.

1972
01:18:34,200 --> 01:18:37,300
And you just finished your
first playable prototype,

1973
01:18:37,300 --> 01:18:38,620
and you did a good test.

1974
01:18:38,620 --> 01:18:40,170
So give yourself a hand.

1975
01:18:40,170 --> 01:18:43,110
[APPLAUSE]

1976
01:18:45,570 --> 01:18:46,380
Let's see.

1977
01:18:46,380 --> 01:18:48,230
Things to keep in mind.

1978
01:18:48,230 --> 01:18:51,400
For those of you who are
making mouse-driven games,

1979
01:18:51,400 --> 01:18:53,500
you want to make sure
that you have enough mice

1980
01:18:53,500 --> 01:18:59,530
to have for your next play test
which will be probably project

1981
01:18:59,530 --> 01:19:01,490
three.

1982
01:19:01,490 --> 01:19:04,560
And you've only got
another five days,

1983
01:19:04,560 --> 01:19:08,500
if I'm not mistaken,
when this is done.

1984
01:19:08,500 --> 01:19:12,210
This is exactly a
two week project.

1985
01:19:12,210 --> 01:19:15,300
So if you haven't
started cutting features,

1986
01:19:15,300 --> 01:19:17,640
this is probably
the time to do it.

1987
01:19:17,640 --> 01:19:19,980
Let's see.

1988
01:19:19,980 --> 01:19:22,620
Something that, of course,
we did notice and you all

1989
01:19:22,620 --> 01:19:26,040
felt it was that there weren't
enough stations for playing

1990
01:19:26,040 --> 01:19:27,110
the games.

1991
01:19:27,110 --> 01:19:30,840
For future play tests,
for project three onwards,

1992
01:19:30,840 --> 01:19:37,880
we are going to expect at least
half as many game computers

1993
01:19:37,880 --> 01:19:41,030
as you have at the
average team size.

1994
01:19:41,030 --> 01:19:43,640
So I'm not sure whether
we're doing with teams of six

1995
01:19:43,640 --> 01:19:45,310
or teams of eight.

1996
01:19:45,310 --> 01:19:49,070
We're doing teams of six
for project three, which

1997
01:19:49,070 --> 01:19:50,610
means each team
should be prepared

1998
01:19:50,610 --> 01:19:54,020
to have at least three computers
running with the games.

1999
01:19:54,020 --> 01:19:56,540
Something else that
we're also going

2000
01:19:56,540 --> 01:20:00,430
to make it a lot clearer
where the game stations are

2001
01:20:00,430 --> 01:20:04,050
is that the computers that are
not being used for game play

2002
01:20:04,050 --> 01:20:05,400
should actually just be closed.

2003
01:20:05,400 --> 01:20:08,275
Take notes using paper, because
it gets confusing to figure out

2004
01:20:08,275 --> 01:20:10,900
which are the game stations and
which ones are just like people

2005
01:20:10,900 --> 01:20:14,440
taking notes and which ones are
the ones where people actually

2006
01:20:14,440 --> 01:20:17,524
coding on the sly trying to
fix last minute problems.

2007
01:20:17,524 --> 01:20:19,940
You really should have all
that sorted out before the play

2008
01:20:19,940 --> 01:20:21,520
test, not during the play test.

2009
01:20:22,620 --> 01:20:24,880
So next time when
we're doing play test,

2010
01:20:24,880 --> 01:20:28,470
just make sure that half
your team, basically,

2011
01:20:28,470 --> 01:20:30,760
is ready to go with the game.

2012
01:20:30,760 --> 01:20:32,370
And I would suggest
more than half

2013
01:20:32,370 --> 01:20:35,100
your team should be able to
run the code, just in case

2014
01:20:35,100 --> 01:20:37,710
someone falls sick, right?

2015
01:20:37,710 --> 01:20:41,180
And doesn't bring their
computer or their mouse.

2016
01:20:41,180 --> 01:20:45,050
Rest of the class is for you
to work on your own team.

2017
01:20:45,050 --> 01:20:48,410
If anyone hasn't signed
into the attendance sheet,

2018
01:20:48,410 --> 01:20:49,860
we still have it up here.

2019
01:20:49,860 --> 01:20:55,810
And I want to have everyone
give the folks from Riot

2020
01:20:55,810 --> 01:20:58,320
a big hand for coming and
giving their feedback.

2021
01:20:58,320 --> 01:21:00,260
[APPLAUSE]

2022
01:21:00,260 --> 01:21:02,685
Thank you.

2023
01:21:02,685 --> 01:21:04,150
GENEVIEVE CONLEY: [INAUDIBLE].

2024
01:21:04,150 --> 01:21:04,983
PHILIP TAN: Oh, yes.

2025
01:21:04,983 --> 01:21:07,705
They have lots of information
about internships.

2026
01:21:07,705 --> 01:21:08,205
Oops.

2027
01:21:08,205 --> 01:21:08,770
Oh, whoa.

2028
01:21:08,770 --> 01:21:12,720
A very, very large stack of
documents and positions that

2029
01:21:12,720 --> 01:21:13,430
are available.

2030
01:21:13,430 --> 01:21:14,940
And a deadline,
October 15th, which

2031
01:21:14,940 --> 01:21:17,730
is not that long from now.

2032
01:21:17,730 --> 01:21:21,930
So definitely check those
out as soon as you can.

2033
01:21:21,930 --> 01:21:22,670
Let's see.

2034
01:21:22,670 --> 01:21:24,690
Actually, while
we're on internships,

2035
01:21:24,690 --> 01:21:28,400
there's one other thing
that I wanted to mention.

2036
01:21:28,400 --> 01:21:32,510
I sent out a message for
Europe over in AeroAstro.

2037
01:21:32,510 --> 01:21:36,260
They are looking
for game developers.

2038
01:21:36,260 --> 01:21:38,410
Even better if you
also know MATLAB,

2039
01:21:38,410 --> 01:21:41,530
but they just want
game developers

2040
01:21:41,530 --> 01:21:43,710
for a high school
science competition

2041
01:21:43,710 --> 01:21:46,139
that they're running.

2042
01:21:46,139 --> 01:21:47,180
It's like first robotics.

2043
01:21:47,180 --> 01:21:50,180
It's a robotics competition,
only in zero gravity.

2044
01:21:50,180 --> 01:21:53,320
So if that sounds cool,
you should definitely

2045
01:21:53,320 --> 01:21:56,630
respond to them quickly because
Europe deadlines are coming up.

2046
01:21:56,630 --> 01:22:01,600
And that's it.

2047
01:22:01,600 --> 01:22:03,450
See you [INAUDIBLE].