1
00:00:00,090 --> 00:00:02,500
The following content is
provided under a Creative

2
00:00:02,500 --> 00:00:04,019
Commons license.

3
00:00:04,019 --> 00:00:06,360
Your support will help
MIT OpenCourseWare

4
00:00:06,360 --> 00:00:10,730
continue to offer high quality
educational resources for free.

5
00:00:10,730 --> 00:00:13,330
To make a donation or
view additional materials

6
00:00:13,330 --> 00:00:17,236
from hundreds of MIT courses,
visit MIT OpenCourseWare

7
00:00:17,236 --> 00:00:17,861
at ocw.mit.edu.

8
00:00:21,800 --> 00:00:25,676
PROFESSOR: So today
we have from EA,

9
00:00:25,676 --> 00:00:29,380
Timothy Cowan's going to
give a lecture on development

10
00:00:29,380 --> 00:00:33,622
best practices from
Electronic Arts.

11
00:00:33,622 --> 00:00:35,080
This is about the
time of the class

12
00:00:35,080 --> 00:00:38,240
where we start doing
hour long lectures

13
00:00:38,240 --> 00:00:41,780
and then the remaining two hours
is open for your work time.

14
00:00:41,780 --> 00:00:45,880
It's really going to start
being like that probably not

15
00:00:45,880 --> 00:00:47,910
next Wednesday, but
the following Monday

16
00:00:47,910 --> 00:00:49,909
is when that's going to
be a pretty stable time.

17
00:00:49,909 --> 00:00:51,720
So when you're working
on project four,

18
00:00:51,720 --> 00:00:55,967
you're going to have a lot more
in class time to work in teams.

19
00:00:55,967 --> 00:00:58,300
Hopefully you've been taking
advantage of the class time

20
00:00:58,300 --> 00:01:00,960
to work in your teams
for project three.

21
00:01:00,960 --> 00:01:03,530
Today we are doing play testing.

22
00:01:03,530 --> 00:01:06,344
Has everybody already
prepared for what

23
00:01:06,344 --> 00:01:08,260
they're going to do for
their play test today?

24
00:01:08,260 --> 00:01:10,800
Do you already have
questionnaire, survey, observer

25
00:01:10,800 --> 00:01:11,805
notes, things like that?

26
00:01:14,380 --> 00:01:15,730
I'm seeing some blank faces.

27
00:01:15,730 --> 00:01:16,430
That's OK.

28
00:01:16,430 --> 00:01:18,610
We're going to give you
15 minutes to set that up.

29
00:01:18,610 --> 00:01:21,560
So we're going to do
lecture, take a short break,

30
00:01:21,560 --> 00:01:25,400
then you'll get 15 minutes to
prepare for the focus test.

31
00:01:25,400 --> 00:01:28,120
We're asking you to set
up three workstations

32
00:01:28,120 --> 00:01:29,850
for the test at all times.

33
00:01:29,850 --> 00:01:33,150
I'll go through the details
when we start that up.

34
00:01:33,150 --> 00:01:36,310
But you'll get a little bit of
time to set up for play test.

35
00:01:36,310 --> 00:01:38,660
Remember, we're asking
for two focus test

36
00:01:38,660 --> 00:01:40,295
reports for project three.

37
00:01:40,295 --> 00:01:41,920
This is going to
count for one of them.

38
00:01:41,920 --> 00:01:44,180
The other one you need
to do on your own time

39
00:01:44,180 --> 00:01:47,210
or have already done.

40
00:01:47,210 --> 00:01:50,860
Grading is practically
finished for project two.

41
00:01:50,860 --> 00:01:54,290
And now it's just entering
it all into the Stellar form,

42
00:01:54,290 --> 00:01:57,340
and you'll have all
that by tomorrow night.

43
00:01:57,340 --> 00:01:59,610
What I can say just in
general for grading,

44
00:01:59,610 --> 00:02:01,570
project two looked pretty good.

45
00:02:01,570 --> 00:02:06,730
Most of the games completed
all of the requirements.

46
00:02:06,730 --> 00:02:08,440
The main requirements
that we saw

47
00:02:08,440 --> 00:02:10,280
that were lacking
in some of the games

48
00:02:10,280 --> 00:02:12,990
were really the
legal requirements.

49
00:02:12,990 --> 00:02:16,370
Putting your credits, putting
your names on the game.

50
00:02:16,370 --> 00:02:17,300
We asked for that.

51
00:02:17,300 --> 00:02:19,540
So please make sure you're
doing for project three.

52
00:02:19,540 --> 00:02:21,510
Even if it's not
in the game itself,

53
00:02:21,510 --> 00:02:26,314
it can be on the full sheet,
the full HTML code for the game.

54
00:02:26,314 --> 00:02:28,480
So make sure you're putting
your names on your games

55
00:02:28,480 --> 00:02:30,660
when you're turning them in.

56
00:02:30,660 --> 00:02:33,060
And also if you're
using any outside code,

57
00:02:33,060 --> 00:02:35,260
any outside assets,
that should all

58
00:02:35,260 --> 00:02:37,440
be listed as where
it's coming from

59
00:02:37,440 --> 00:02:40,910
and what license
you have for that.

60
00:02:40,910 --> 00:02:43,740
Phaser is MIT licensed,
so there wasn't

61
00:02:43,740 --> 00:02:44,870
anything special for there.

62
00:02:44,870 --> 00:02:47,130
But if you're using Unity,
take a look at Unity's license

63
00:02:47,130 --> 00:02:47,620
requirements.

64
00:02:47,620 --> 00:02:48,970
Make sure you're
fulfilling them.

65
00:02:48,970 --> 00:02:51,261
If you're using hacks or
Flixel, make sure your filling

66
00:02:51,261 --> 00:02:52,960
those requirements.

67
00:02:52,960 --> 00:02:55,570
I believe we already gave
you some comments on the post

68
00:02:55,570 --> 00:02:56,070
forms.

69
00:02:56,070 --> 00:02:58,460
But for the most part,
those also looked good.

70
00:02:58,460 --> 00:03:02,420
For project three, here's the
cheat sheet for product three's

71
00:03:02,420 --> 00:03:03,660
postmortems.

72
00:03:03,660 --> 00:03:05,800
You've got a page to
write a postmortem.

73
00:03:05,800 --> 00:03:09,500
Make half of it about your
team practices and processes.

74
00:03:09,500 --> 00:03:12,250
The other half about
the game and the design.

75
00:03:12,250 --> 00:03:14,490
That's really what we're
looking for is understanding

76
00:03:14,490 --> 00:03:17,370
how your team organized itself,
what communication problems

77
00:03:17,370 --> 00:03:21,090
you had, how you solved them,
and even more importantly, what

78
00:03:21,090 --> 00:03:22,590
are you going to
do in project four.

79
00:03:22,590 --> 00:03:23,540
Because in product
four, you're going

80
00:03:23,540 --> 00:03:25,250
to have eight
people on your team

81
00:03:25,250 --> 00:03:27,780
and problems are going
to show up more often.

82
00:03:27,780 --> 00:03:31,350
So hopefully you're coming
up with ideas and ways

83
00:03:31,350 --> 00:03:34,900
to be prepared for
those kind of things.

84
00:03:34,900 --> 00:03:35,970
Any other comments?

85
00:03:35,970 --> 00:03:40,870
Otherwise for the other
turn ins for product two,

86
00:03:40,870 --> 00:03:42,510
task lists were good.

87
00:03:42,510 --> 00:03:45,420
Backlogs were acceptable.

88
00:03:45,420 --> 00:03:47,790
The focus test reports
for the most acceptable.

89
00:03:47,790 --> 00:03:50,510
We give you some comments
about your focus test reports.

90
00:03:50,510 --> 00:03:54,640
We gave you some comments
about your design change logs.

91
00:03:54,640 --> 00:03:57,110
We also are giving you
a big chunk of comments

92
00:03:57,110 --> 00:03:58,480
about the game design.

93
00:03:58,480 --> 00:04:01,470
It's going to be labeled
not part of the grade.

94
00:04:01,470 --> 00:04:03,720
So like I said, it's
not a design class,

95
00:04:03,720 --> 00:04:06,570
but we do want to give you
some feedback on your designs.

96
00:04:06,570 --> 00:04:11,415
So we can help you out
with future assignments.

97
00:04:11,415 --> 00:04:11,960
That's it.

98
00:04:11,960 --> 00:04:15,680
Any questions about anything?

99
00:04:15,680 --> 00:04:16,540
Great.

100
00:04:16,540 --> 00:04:19,490
So I'm just going to
hand it off to Tim.

101
00:04:19,490 --> 00:04:21,220
TIM COWAN: Great.

102
00:04:21,220 --> 00:04:23,290
So can everybody hear me?

103
00:04:23,290 --> 00:04:24,380
Sound good?

104
00:04:24,380 --> 00:04:28,050
You can just raise your hand in
the back if you can't hear me.

105
00:04:28,050 --> 00:04:28,750
Good?

106
00:04:28,750 --> 00:04:30,576
All right.

107
00:04:30,576 --> 00:04:32,950
So let me start off a little
bit with just introductions.

108
00:04:32,950 --> 00:04:37,960
Who I am, my background, where I
came from, why I'm here, maybe.

109
00:04:37,960 --> 00:04:40,560
I'm actually originally
from Oklahoma.

110
00:04:40,560 --> 00:04:42,520
I was born in Tulsa, Oklahoma.

111
00:04:42,520 --> 00:04:44,970
And went to Oklahoma
State University.

112
00:04:44,970 --> 00:04:47,930
And going to
university, I actually

113
00:04:47,930 --> 00:04:49,340
had no idea what I wanted to do.

114
00:04:49,340 --> 00:04:52,640
Kind of just sort of landed at
one of the in state tuitions

115
00:04:52,640 --> 00:04:53,360
for university.

116
00:04:53,360 --> 00:04:56,550
But I knew my dad was an
engineer, a programmer

117
00:04:56,550 --> 00:04:58,740
specifically, and I
kind of thought that I

118
00:04:58,740 --> 00:05:01,091
wanted to go do what he did.

119
00:05:01,091 --> 00:05:02,590
So I went into
university, was going

120
00:05:02,590 --> 00:05:04,905
to go be trained
to be a programmer.

121
00:05:04,905 --> 00:05:09,642
He actually gave me advice to go
work on more pure engineering.

122
00:05:09,642 --> 00:05:11,100
And so for my
undergraduate degree,

123
00:05:11,100 --> 00:05:14,600
I was actually focused on
mechanical engineering.

124
00:05:14,600 --> 00:05:17,350
And I stayed at Oklahoma
State University

125
00:05:17,350 --> 00:05:21,590
and went on to
aerospace engineering.

126
00:05:21,590 --> 00:05:23,500
Did graduate school.

127
00:05:23,500 --> 00:05:26,800
Was in pursuit of a Ph.D.
with ultimately the goal

128
00:05:26,800 --> 00:05:29,400
I wanted to be an astronaut.

129
00:05:29,400 --> 00:05:30,770
I didn't make it there.

130
00:05:30,770 --> 00:05:32,760
I made video games today.

131
00:05:32,760 --> 00:05:35,190
But I did get a chance to
through graduate school

132
00:05:35,190 --> 00:05:37,070
work on research for
Dryden Flight Research

133
00:05:37,070 --> 00:05:41,110
Center in California and
actually spent about 10 months

134
00:05:41,110 --> 00:05:44,940
there as a visiting researcher
while I was working on my Ph.D.

135
00:05:44,940 --> 00:05:49,900
And what I did for them was what
I'd say were flight simulators.

136
00:05:49,900 --> 00:05:51,700
But not that kind
of flight simulator.

137
00:05:51,700 --> 00:05:54,590
It was more this kind
of flight simulator.

138
00:05:54,590 --> 00:05:56,740
Computer simulation of aircraft.

139
00:05:56,740 --> 00:05:59,810
I did a lot of both the
simulation of the aircraft

140
00:05:59,810 --> 00:06:02,179
and generating visual
representations

141
00:06:02,179 --> 00:06:02,970
of the simulations.

142
00:06:05,560 --> 00:06:11,770
And ultimately out of all that
work I did finish my Ph.D.

143
00:06:11,770 --> 00:06:16,300
and sort of officially
became a rocket scientist.

144
00:06:16,300 --> 00:06:18,820
And so then I had to decide
what I wanted to go do.

145
00:06:18,820 --> 00:06:22,870
And my plan really was
to go work at Lockheed,

146
00:06:22,870 --> 00:06:26,800
design aircraft, work on
flight simulators for them.

147
00:06:26,800 --> 00:06:30,420
That was basically my life's
mission coming out of college.

148
00:06:30,420 --> 00:06:32,230
But that didn't
quite happen either

149
00:06:32,230 --> 00:06:35,920
and I ended up at EA Sports.

150
00:06:35,920 --> 00:06:38,930
So instead of working on
flight simulators, what

151
00:06:38,930 --> 00:06:41,710
I've done for the
last 14 years is work

152
00:06:41,710 --> 00:06:44,910
on NFL football simulators.

153
00:06:44,910 --> 00:06:47,890
A recruiter from EA
just basically blind

154
00:06:47,890 --> 00:06:51,850
contacted me from my resume
posting on Monster.com

155
00:06:51,850 --> 00:06:54,060
and they brought me
down to Austin, Texas

156
00:06:54,060 --> 00:06:55,630
to work on my first
video game, which

157
00:06:55,630 --> 00:06:59,730
was Madden NFL 2002 for the PC.

158
00:06:59,730 --> 00:07:02,350
There was a small team in
Austin of about six people

159
00:07:02,350 --> 00:07:05,200
that built this product
off of a shared code

160
00:07:05,200 --> 00:07:09,680
base from the PlayStation
to Xbox and Game Cube.

161
00:07:09,680 --> 00:07:12,190
And I spent about
18 months in Austin.

162
00:07:12,190 --> 00:07:15,164
Built two versions of
Madden PC in that time,

163
00:07:15,164 --> 00:07:16,580
until they eventually
moved me out

164
00:07:16,580 --> 00:07:20,050
to Florida, which was where
Tiburon Entertainment was

165
00:07:20,050 --> 00:07:22,855
located, one of the
studios that's part of EA

166
00:07:22,855 --> 00:07:25,240
and part of EA Sports.

167
00:07:25,240 --> 00:07:27,710
And I've been there,
really been with Tiburon

168
00:07:27,710 --> 00:07:30,590
for I guess it's
counting 13 years now.

169
00:07:30,590 --> 00:07:33,500
And in that span
one of the things

170
00:07:33,500 --> 00:07:36,410
I'm most proud of is I
built 14 versions of Madden

171
00:07:36,410 --> 00:07:38,550
in that time.

172
00:07:38,550 --> 00:07:41,820
And I can say, with the
exception of Madden 15,

173
00:07:41,820 --> 00:07:45,470
because my roles changed, I've
contributed at least one line

174
00:07:45,470 --> 00:07:49,540
of code to 13 versions of
Madden over the last 13 years.

175
00:07:52,127 --> 00:07:53,710
Things are pretty
different at Tiburon

176
00:07:53,710 --> 00:07:55,085
from when I started in 2001.

177
00:07:55,085 --> 00:07:57,460
There were about 200 people
at the studio when I started.

178
00:07:57,460 --> 00:08:00,870
There are 700 at
our location today.

179
00:08:00,870 --> 00:08:03,060
We work on three
different products.

180
00:08:03,060 --> 00:08:05,250
And I'm the group technical
director for the studio.

181
00:08:05,250 --> 00:08:09,590
So I help make sure that all
three of the products we build

182
00:08:09,590 --> 00:08:13,160
succeeds and sort of monitoring
engineering and technology

183
00:08:13,160 --> 00:08:15,740
development for the studio.

184
00:08:15,740 --> 00:08:18,520
Our three products
are obviously Madden.

185
00:08:18,520 --> 00:08:22,030
We build NBA Live
and we're working

186
00:08:22,030 --> 00:08:24,830
on the next generation
of PGA Tour Golf, which

187
00:08:24,830 --> 00:08:27,880
will be out this March.

188
00:08:27,880 --> 00:08:29,446
So that's my background.

189
00:08:29,446 --> 00:08:30,570
A little bit about my role.

190
00:08:30,570 --> 00:08:33,169
It is very much it is an
engineering leadership role

191
00:08:33,169 --> 00:08:35,480
as a group technical director.

192
00:08:35,480 --> 00:08:37,840
And I very much don't
write much code.

193
00:08:37,840 --> 00:08:39,840
In fact, I kind of joke
with the other engineers

194
00:08:39,840 --> 00:08:42,030
that the coding that I do
is PowerPoint these days.

195
00:08:45,700 --> 00:08:47,430
Let's talk a little
bit about Madden.

196
00:08:47,430 --> 00:08:49,650
Set a little bit of
context about some

197
00:08:49,650 --> 00:08:52,170
of the philosophy
and best practices

198
00:08:52,170 --> 00:08:56,510
that I hope to a kind
of pass along today.

199
00:08:56,510 --> 00:09:00,365
Madden in North America
has been a we'll call it

200
00:09:00,365 --> 00:09:03,910
a top five product for at least
the last 15 years, typically

201
00:09:03,910 --> 00:09:08,430
is a top three product
in North America.

202
00:09:08,430 --> 00:09:12,500
Generally the first person
shooters are the big sellers,

203
00:09:12,500 --> 00:09:14,110
especially in North America.

204
00:09:14,110 --> 00:09:17,100
Sometimes you've got
some of the action games

205
00:09:17,100 --> 00:09:18,475
like Grand Theft Auto.

206
00:09:18,475 --> 00:09:20,850
But depending on what's being
released at any given year,

207
00:09:20,850 --> 00:09:23,930
Madden's always a number two,
number three in North American

208
00:09:23,930 --> 00:09:25,500
sales.

209
00:09:25,500 --> 00:09:29,805
And you can see FIFA top
10 North American sales.

210
00:09:29,805 --> 00:09:31,680
They're a different sort
of a sports product.

211
00:09:31,680 --> 00:09:34,750
That is the big sports product
but they have global appeal,

212
00:09:34,750 --> 00:09:36,650
because the sport
has global appeal

213
00:09:36,650 --> 00:09:39,770
and it sells in a lot of
markets that Madden does not.

214
00:09:39,770 --> 00:09:46,430
But as a North American company,
Electronic Arts, the number two

215
00:09:46,430 --> 00:09:47,680
title in North America.

216
00:09:47,680 --> 00:09:49,930
In fact, on that last
chart, the number one title

217
00:09:49,930 --> 00:09:53,970
in North America that
EA sold is Madden.

218
00:09:53,970 --> 00:09:55,590
And it's very important to EA.

219
00:09:55,590 --> 00:09:58,020
It's very important
to EA stock price.

220
00:09:58,020 --> 00:10:03,760
If you look at any sort of
financial reports or just

221
00:10:03,760 --> 00:10:07,090
even news briefs on
financial listings for EA,

222
00:10:07,090 --> 00:10:10,460
you'll often see that the little
blurb at the bottom where they

223
00:10:10,460 --> 00:10:13,330
kind of describe who
Electronic Arts is, Madden's

224
00:10:13,330 --> 00:10:15,740
almost always listed as
Electronic Arts, developer

225
00:10:15,740 --> 00:10:19,650
of products such as
Madden, Battlefield, FIFA.

226
00:10:19,650 --> 00:10:22,400
Madden is very important to EA.

227
00:10:22,400 --> 00:10:24,340
And the success or
failure of that product

228
00:10:24,340 --> 00:10:27,600
is very important to the
public company's stock price.

229
00:10:30,270 --> 00:10:31,930
If you look at
Madden development,

230
00:10:31,930 --> 00:10:33,430
given its importance
to the company,

231
00:10:33,430 --> 00:10:37,360
given the size of its
sales, it's a big team.

232
00:10:37,360 --> 00:10:39,130
There's over 200
developers on staff

233
00:10:39,130 --> 00:10:41,060
that do direct
development on Madden.

234
00:10:41,060 --> 00:10:45,780
That includes engineers,
artists, designers, producers,

235
00:10:45,780 --> 00:10:52,240
product managers, quality
assurance testers.

236
00:10:52,240 --> 00:10:54,006
And there are additional
developers that

237
00:10:54,006 --> 00:10:55,130
contribute through central.

238
00:10:55,130 --> 00:10:57,187
Technology We're a
very large organization

239
00:10:57,187 --> 00:10:59,020
that contributes in a
lot of different ways.

240
00:10:59,020 --> 00:11:02,060
But sort of at our studio
we call it 200 people

241
00:11:02,060 --> 00:11:04,470
doing direct dev on Madden.

242
00:11:04,470 --> 00:11:09,230
10 million lines of
code, give or take a few.

243
00:11:09,230 --> 00:11:10,910
That's physical lines of codes.

244
00:11:10,910 --> 00:11:12,750
So it includes white
space and comments

245
00:11:12,750 --> 00:11:14,260
and those kinds of things.

246
00:11:14,260 --> 00:11:20,420
But a lot, a lot of code to look
at across 150,000 plus files.

247
00:11:20,420 --> 00:11:22,570
And we do development
distributed across four

248
00:11:22,570 --> 00:11:23,990
different locations.

249
00:11:23,990 --> 00:11:26,880
So we've got primary
development in Orlando,

250
00:11:26,880 --> 00:11:28,520
but we also have
remote development

251
00:11:28,520 --> 00:11:33,370
in Austin and a contract company
that we work with Nova Scotia.

252
00:11:33,370 --> 00:11:39,560
And Vancouver is the sibling
studio for Sports, EAC,

253
00:11:39,560 --> 00:11:44,220
who makes FIFA, NHL, USC.

254
00:11:44,220 --> 00:11:46,590
They contribute a lot
of central technology

255
00:11:46,590 --> 00:11:48,754
that helps ship
Madden each year.

256
00:11:48,754 --> 00:11:50,420
So there's really
sort of four locations

257
00:11:50,420 --> 00:11:53,790
that we're distributed across.

258
00:11:53,790 --> 00:11:57,620
Really the message
here is Madden's big.

259
00:11:57,620 --> 00:11:58,960
It's a big seller.

260
00:11:58,960 --> 00:12:00,660
It has big importance.

261
00:12:00,660 --> 00:12:03,380
And it's big large
scale development.

262
00:12:03,380 --> 00:12:06,180
Large teams, large code
bases, large complexity.

263
00:12:09,020 --> 00:12:11,030
And the other
characteristic about

264
00:12:11,030 --> 00:12:15,940
is kind of important
to sort of understand

265
00:12:15,940 --> 00:12:19,100
sort of the nature of this
project, this big project that

266
00:12:19,100 --> 00:12:23,510
also iterates at an iterative
cycle every single year.

267
00:12:23,510 --> 00:12:25,970
So our sports products come
out at the start of the sports

268
00:12:25,970 --> 00:12:28,110
season every single year.

269
00:12:28,110 --> 00:12:33,000
And it's critical that Madden
comes out on its date in August

270
00:12:33,000 --> 00:12:35,070
when the NFL season kicks off.

271
00:12:35,070 --> 00:12:38,020
Our gamers expect it.

272
00:12:38,020 --> 00:12:40,610
Our shareholders
expect it to sell.

273
00:12:40,610 --> 00:12:43,660
And if you miss that
start of the NFL season

274
00:12:43,660 --> 00:12:46,880
and were to, say,
slip your ship date,

275
00:12:46,880 --> 00:12:49,510
that means a lot of lost
opportunity in terms

276
00:12:49,510 --> 00:12:52,250
of delighting our gamers.

277
00:12:52,250 --> 00:12:54,930
That's very different from
other types of game development.

278
00:12:54,930 --> 00:13:02,160
Like certain games, they're
done when they're done.

279
00:13:02,160 --> 00:13:04,850
And even within EA we
do this a little bit.

280
00:13:04,850 --> 00:13:06,790
This was one of
the announcements

281
00:13:06,790 --> 00:13:09,120
we made about Mirrors Edge.

282
00:13:09,120 --> 00:13:10,500
That's not Madden.

283
00:13:10,500 --> 00:13:12,730
Madden can never miss its date.

284
00:13:12,730 --> 00:13:17,317
And it has come out in
August for the last 25 years

285
00:13:17,317 --> 00:13:18,400
consistently and reliably.

286
00:13:21,050 --> 00:13:24,280
So given that context, what I
want to talk about a little bit

287
00:13:24,280 --> 00:13:26,850
first is just some philosophy.

288
00:13:26,850 --> 00:13:30,490
And this is some philosophy I've
come to over the past couple

289
00:13:30,490 --> 00:13:34,490
years about what I think
makes a good engineer at EA

290
00:13:34,490 --> 00:13:37,200
and specifically on Madden
and our sports products.

291
00:13:37,200 --> 00:13:40,240
What some of the mindset
then things that you

292
00:13:40,240 --> 00:13:42,510
may need to break
out of in order

293
00:13:42,510 --> 00:13:46,560
to be successful doing
development on the big title

294
00:13:46,560 --> 00:13:49,072
that's got to come out
on a particular date.

295
00:13:49,072 --> 00:13:52,160
And when you are big
title and a big team

296
00:13:52,160 --> 00:13:56,260
and a complicated project that
has to hit a particular date,

297
00:13:56,260 --> 00:13:57,730
the first thing
that happens is we

298
00:13:57,730 --> 00:14:01,290
go add a lot of really
complicated process.

299
00:14:01,290 --> 00:14:03,160
Because there's a
lot of moving parts.

300
00:14:03,160 --> 00:14:04,680
Everything's got to get tracked.

301
00:14:04,680 --> 00:14:07,540
And if anything
slips, then we're

302
00:14:07,540 --> 00:14:09,950
not going to hit our date.

303
00:14:09,950 --> 00:14:13,730
You add a lot of checklists
and you check things off.

304
00:14:13,730 --> 00:14:15,280
Hopefully get
everything checked off.

305
00:14:17,840 --> 00:14:21,760
But the reality is there's
always a lot of work to do.

306
00:14:21,760 --> 00:14:24,590
We've got 200 plus
people doing a lot

307
00:14:24,590 --> 00:14:27,310
of different types of work and
you only have about one year

308
00:14:27,310 --> 00:14:28,690
to get work done.

309
00:14:28,690 --> 00:14:31,360
And we want to do
and put as much

310
00:14:31,360 --> 00:14:34,640
as we possibly can into that
product every single year.

311
00:14:34,640 --> 00:14:37,470
Because if nothing
else, I know I'm

312
00:14:37,470 --> 00:14:40,090
very sensitive to
our sports products.

313
00:14:40,090 --> 00:14:42,470
Because they come
out every year,

314
00:14:42,470 --> 00:14:46,599
we tend to get beat up a little
bit over just a roster update.

315
00:14:46,599 --> 00:14:47,640
They haven't done enough.

316
00:14:47,640 --> 00:14:50,240
There's not enough improvements
over last year's product.

317
00:14:50,240 --> 00:14:54,857
Well we do a lot of work and
we want to do a lot of work

318
00:14:54,857 --> 00:14:57,440
to make sure that every single
year we're putting out the best

319
00:14:57,440 --> 00:14:59,710
possible product that we can.

320
00:14:59,710 --> 00:15:01,180
And it's not just us.

321
00:15:01,180 --> 00:15:03,714
Also our gamers
really want-- they're

322
00:15:03,714 --> 00:15:04,880
giving us a lot of feedback.

323
00:15:04,880 --> 00:15:07,750
We want this, let's
get this feature in,

324
00:15:07,750 --> 00:15:11,100
Madden would be better
if you just did this.

325
00:15:11,100 --> 00:15:14,630
So there's a lot of work
and our kind of human nature

326
00:15:14,630 --> 00:15:16,670
is let's do it all.

327
00:15:16,670 --> 00:15:18,350
We're going to get
it all done, we're

328
00:15:18,350 --> 00:15:20,070
going to ship an
amazing product,

329
00:15:20,070 --> 00:15:23,046
and we're going to get it
all done on time in August.

330
00:15:23,046 --> 00:15:25,730
Well if you take on too much
and you try to do everything,

331
00:15:25,730 --> 00:15:28,620
disaster happens inevitably.

332
00:15:28,620 --> 00:15:31,200
And one thing we have
to remember in sports

333
00:15:31,200 --> 00:15:34,700
is development's a
multi-year journey.

334
00:15:34,700 --> 00:15:37,510
It's a path.

335
00:15:37,510 --> 00:15:40,290
Most game products,
especially if you're

336
00:15:40,290 --> 00:15:44,660
looking at a open world action
adventure game or first person

337
00:15:44,660 --> 00:15:47,960
shooter, they tend to have
two if not three or four

338
00:15:47,960 --> 00:15:49,397
years of development.

339
00:15:49,397 --> 00:15:51,480
We kind of start to think
about Madden development

340
00:15:51,480 --> 00:15:54,090
in that same way where
it's a three year

341
00:15:54,090 --> 00:15:57,707
process with incremental
releases along the way, where

342
00:15:57,707 --> 00:16:00,040
the game just keeps getting
better and better and better

343
00:16:00,040 --> 00:16:02,680
on a three year roadmap.

344
00:16:02,680 --> 00:16:04,890
And it's important in
this sense because I

345
00:16:04,890 --> 00:16:08,280
think all game makers want
to do everything, want

346
00:16:08,280 --> 00:16:10,490
to make the game as
amazing as they can.

347
00:16:10,490 --> 00:16:14,530
But in sports, you really
have to embrace iteration.

348
00:16:14,530 --> 00:16:18,240
And that means kind of finding
the right balance between doing

349
00:16:18,240 --> 00:16:21,370
everything, doing
everything right,

350
00:16:21,370 --> 00:16:24,060
and especially if you're
an engineer remembering

351
00:16:24,060 --> 00:16:27,290
that sometimes you have
to balance between getting

352
00:16:27,290 --> 00:16:30,670
everything done right and maybe
using some duct tape and kind

353
00:16:30,670 --> 00:16:32,590
of hacking in a quick solution.

354
00:16:32,590 --> 00:16:35,040
Because you've got
three years to make sure

355
00:16:35,040 --> 00:16:38,910
that you're getting things done
on that road that you're on.

356
00:16:42,430 --> 00:16:45,310
So another idea I'd like to
talk a little bit about, again,

357
00:16:45,310 --> 00:16:49,429
just more philosophy
about engineering.

358
00:16:49,429 --> 00:16:51,220
I talk about the
difference between science

359
00:16:51,220 --> 00:16:52,890
and engineering.

360
00:16:52,890 --> 00:16:56,880
And I'm going to sort of give
the extreme examples and maybe

361
00:16:56,880 --> 00:16:58,585
overgeneralize things.

362
00:16:58,585 --> 00:17:01,090
But one of things I talk
about to my engineers

363
00:17:01,090 --> 00:17:05,630
is, if you look at science,
science at its core

364
00:17:05,630 --> 00:17:08,319
is about the pursuit
of knowledge.

365
00:17:08,319 --> 00:17:10,430
The idea is king.

366
00:17:10,430 --> 00:17:12,731
You're going to try to
figure out how things work.

367
00:17:12,731 --> 00:17:15,230
You don't know what you're going
to do once you find it out.

368
00:17:15,230 --> 00:17:18,300
But just the fact that
I know how that works,

369
00:17:18,300 --> 00:17:24,420
I know this idea, that's
what's most important.

370
00:17:24,420 --> 00:17:28,970
Engineering, on the other
hand, if sort of like the best

371
00:17:28,970 --> 00:17:31,360
engineer, if you're just the
extreme engineer, the thing

372
00:17:31,360 --> 00:17:33,850
you get the most
excitement out of his

373
00:17:33,850 --> 00:17:35,690
taking that knowledge
from the scientists

374
00:17:35,690 --> 00:17:39,710
and figuring out how to apply
it to solve practical problems.

375
00:17:39,710 --> 00:17:42,740
And cost effective
solutions is definitely

376
00:17:42,740 --> 00:17:46,270
one of the keys in terms of
that well rounded engineer that

377
00:17:46,270 --> 00:17:48,760
goes in and is figuring
out how to solve

378
00:17:48,760 --> 00:17:51,975
a very practical
problem based on a body

379
00:17:51,975 --> 00:17:52,975
of scientific knowledge.

380
00:17:55,580 --> 00:17:57,150
It's really important
for engineers

381
00:17:57,150 --> 00:18:00,900
remember that there are
solutions to even the hardest

382
00:18:00,900 --> 00:18:02,800
problems.

383
00:18:02,800 --> 00:18:05,360
If cost is a factor,
you often find

384
00:18:05,360 --> 00:18:10,030
that engineers will tend to
say this type of problem's

385
00:18:10,030 --> 00:18:12,540
not possible.

386
00:18:12,540 --> 00:18:15,650
But what they really mean is
it's very, very expensive.

387
00:18:15,650 --> 00:18:17,660
But engineers need to
really be focused on,

388
00:18:17,660 --> 00:18:19,740
how do you break down
a problem and come up

389
00:18:19,740 --> 00:18:21,110
with a solution to that problem?

390
00:18:21,110 --> 00:18:25,260
And sometimes it's duct tape.

391
00:18:25,260 --> 00:18:27,740
And for the Star
Trek fans out there,

392
00:18:27,740 --> 00:18:30,595
remember there are no unwinnable
scenarios in engineering.

393
00:18:34,440 --> 00:18:35,785
There is no Mission Impossible.

394
00:18:40,880 --> 00:18:43,540
I've had engineers on
Madden in particular

395
00:18:43,540 --> 00:18:47,660
where Madden in the
early Xbox 360 generation

396
00:18:47,660 --> 00:18:51,087
actually ran at 30
frames per second.

397
00:18:51,087 --> 00:18:52,670
And there were a few
years there where

398
00:18:52,670 --> 00:18:54,820
we were told it was
impossible for Madden

399
00:18:54,820 --> 00:18:57,640
to run at 60 frames per second.

400
00:18:57,640 --> 00:19:00,410
And so in a leadership
position, a lot of your job

401
00:19:00,410 --> 00:19:04,220
is to kind of break through
that this is impossible

402
00:19:04,220 --> 00:19:06,080
and start to ask
questions on how can we

403
00:19:06,080 --> 00:19:09,180
break this problem down
and come up with some cost

404
00:19:09,180 --> 00:19:10,850
effective solutions
that will actually

405
00:19:10,850 --> 00:19:12,820
allow us to solve this problem.

406
00:19:15,840 --> 00:19:18,360
And at the end of
the day, you're

407
00:19:18,360 --> 00:19:20,790
not going to solve problems
if you don't attack them

408
00:19:20,790 --> 00:19:22,450
with enthusiasm.

409
00:19:22,450 --> 00:19:24,930
And so my
encouragement is really

410
00:19:24,930 --> 00:19:26,990
to challenge the impossible.

411
00:19:26,990 --> 00:19:28,420
Everything is possible.

412
00:19:28,420 --> 00:19:31,276
It maybe might not
be possible today.

413
00:19:31,276 --> 00:19:32,900
It might not be
possible with the money

414
00:19:32,900 --> 00:19:34,560
you've got in your wallet.

415
00:19:34,560 --> 00:19:39,230
But given enough time and
money, we can do anything.

416
00:19:39,230 --> 00:19:41,990
And every problem can be solved.

417
00:19:41,990 --> 00:19:44,710
At least at the
core, a good engineer

418
00:19:44,710 --> 00:19:45,990
really needs to believe that.

419
00:19:49,940 --> 00:19:52,210
So let's go on next idea.

420
00:19:52,210 --> 00:19:56,220
And I want to dispel the
myth of perfect code.

421
00:19:56,220 --> 00:19:57,920
Because this is
one of those things

422
00:19:57,920 --> 00:20:02,510
that engineers coming into EA,
really coming into any job,

423
00:20:02,510 --> 00:20:05,300
have this myth that
there is perfect code.

424
00:20:05,300 --> 00:20:09,100
Because engineers
love perfect code.

425
00:20:09,100 --> 00:20:11,200
And they really love to
argue about perfect code.

426
00:20:13,800 --> 00:20:16,200
Even the best engineers
in the world, this

427
00:20:16,200 --> 00:20:20,610
is ingrained into every
engineer that not only is there

428
00:20:20,610 --> 00:20:23,200
perfect code, I know what
that perfect code looks like

429
00:20:23,200 --> 00:20:24,700
and I'm the only
one that writes it.

430
00:20:27,220 --> 00:20:31,900
But the reality is,
mistakes are always made.

431
00:20:31,900 --> 00:20:35,385
It tends to be the first
time you write code,

432
00:20:35,385 --> 00:20:36,760
especially if it's
the first time

433
00:20:36,760 --> 00:20:38,460
you've written code for
a problem you've ever

434
00:20:38,460 --> 00:20:39,110
solved before.

435
00:20:39,110 --> 00:20:41,571
You're going to make mistakes.

436
00:20:41,571 --> 00:20:43,930
And I do very much
fundamentally believe

437
00:20:43,930 --> 00:20:48,800
that kind of at the core
inside that you learn more

438
00:20:48,800 --> 00:20:50,420
from mistakes.

439
00:20:50,420 --> 00:20:52,820
Hopefully you guys
have heard that before.

440
00:20:52,820 --> 00:20:55,460
And if you never make any
mistakes, you're not learning.

441
00:20:55,460 --> 00:20:59,530
And if you're never taking
on really big potentially

442
00:20:59,530 --> 00:21:02,170
impossible challenges,
you're not learning.

443
00:21:02,170 --> 00:21:04,750
You're not stretching
yourself, you're not growing.

444
00:21:04,750 --> 00:21:07,199
So it's important to remember
that mistakes will be made.

445
00:21:07,199 --> 00:21:09,740
You're going to learn from those
mistakes, and you know what?

446
00:21:09,740 --> 00:21:14,395
Your code will get better
as you make those mistakes

447
00:21:14,395 --> 00:21:16,050
and refine it.

448
00:21:16,050 --> 00:21:20,630
So remember, there is a
software development life cycle.

449
00:21:20,630 --> 00:21:22,340
And in fact, a lot
of times when you're

450
00:21:22,340 --> 00:21:25,020
arguing about perfect code,
it's very early on in the cycle

451
00:21:25,020 --> 00:21:26,350
and you don't even
know what you don't

452
00:21:26,350 --> 00:21:28,700
know because that code hasn't
been through that software

453
00:21:28,700 --> 00:21:31,570
development life cycle.

454
00:21:31,570 --> 00:21:35,290
There's almost no guarantee that
there's no bugs in this code

455
00:21:35,290 --> 00:21:36,210
that you're writing.

456
00:21:36,210 --> 00:21:38,668
And it really isn't until you
make it through that software

457
00:21:38,668 --> 00:21:43,390
life cycle that you're going to
understand what those bugs are.

458
00:21:43,390 --> 00:21:45,270
Towards the end of
the project where

459
00:21:45,270 --> 00:21:47,460
you have this
perfect code and all

460
00:21:47,460 --> 00:21:49,970
of a sudden now it's been
all the way through testing

461
00:21:49,970 --> 00:21:55,760
and certification and now you
see, oops, I made a mistake.

462
00:21:55,760 --> 00:22:00,120
And when you tie that into
our sports iterative cycle

463
00:22:00,120 --> 00:22:03,540
where we're embracing iteration,
taking on the impossible,

464
00:22:03,540 --> 00:22:06,170
one of the big things that
try to impress on people

465
00:22:06,170 --> 00:22:10,140
is try to value less
the perfect code

466
00:22:10,140 --> 00:22:11,857
and start to think
about proven code.

467
00:22:11,857 --> 00:22:14,440
Proven code that's been through
that software development life

468
00:22:14,440 --> 00:22:17,030
cycle, has had the
bugs shaken out of it.

469
00:22:17,030 --> 00:22:19,340
Even if you didn't
write it, that code

470
00:22:19,340 --> 00:22:23,384
is proven and will have less
bugs typically than code

471
00:22:23,384 --> 00:22:24,675
that you go right from scratch.

472
00:22:29,120 --> 00:22:31,800
And then the next idea,
and this is the last one

473
00:22:31,800 --> 00:22:34,360
before we can get
onto the other section

474
00:22:34,360 --> 00:22:39,340
and get out of the philosophy
stuff, is on a big team,

475
00:22:39,340 --> 00:22:42,080
on that iterative cycle
there's a lot of tasks.

476
00:22:42,080 --> 00:22:43,980
There's a lot of
work to get done.

477
00:22:43,980 --> 00:22:45,840
Everybody's given
their tasks and they

478
00:22:45,840 --> 00:22:48,060
have to finish their tasks.

479
00:22:48,060 --> 00:22:52,300
They've all kind of got this
little piece of a puzzle

480
00:22:52,300 --> 00:22:53,970
and they own their
piece of the puzzle.

481
00:22:53,970 --> 00:22:55,636
They've been given
this task and they're

482
00:22:55,636 --> 00:22:57,470
going to get that piece done.

483
00:22:57,470 --> 00:23:00,200
And it's going to be perfect
given the context that they've

484
00:23:00,200 --> 00:23:03,170
been given on that task.

485
00:23:03,170 --> 00:23:06,000
The trick is there's
a lot of dependencies.

486
00:23:06,000 --> 00:23:07,740
And it's often very
hard to go see,

487
00:23:07,740 --> 00:23:09,130
how do all those
pieces together?

488
00:23:11,929 --> 00:23:13,720
And when you've got
this kind of situation,

489
00:23:13,720 --> 00:23:17,900
you start to end up with you've
got these features, which are

490
00:23:17,900 --> 00:23:19,540
actually a collection of tasks.

491
00:23:19,540 --> 00:23:21,490
And those features don't
really have an owner.

492
00:23:21,490 --> 00:23:23,550
All the tasks have an owner.

493
00:23:23,550 --> 00:23:25,300
Someone's doing the tasks.

494
00:23:25,300 --> 00:23:26,559
But what about the feature?

495
00:23:26,559 --> 00:23:27,850
Does the feature have an owner?

496
00:23:27,850 --> 00:23:30,260
Does anybody know
how all those pieces

497
00:23:30,260 --> 00:23:32,540
are going to fit together?

498
00:23:32,540 --> 00:23:35,210
And if you don't have
someone owning that feature,

499
00:23:35,210 --> 00:23:38,880
you end up with
unfinished projects.

500
00:23:38,880 --> 00:23:41,810
Or potentially you
end up with not quite

501
00:23:41,810 --> 00:23:44,290
what you were looking for
because you didn't really

502
00:23:44,290 --> 00:23:47,162
understand from this task
how it all fit together

503
00:23:47,162 --> 00:23:49,120
and was going to ultimately
deliver whatever it

504
00:23:49,120 --> 00:23:53,370
was that we were looking for.

505
00:23:53,370 --> 00:23:55,490
And so it's super
important when you

506
00:23:55,490 --> 00:23:57,500
get immersed into
a really big team

507
00:23:57,500 --> 00:23:59,000
and you're hand and
a bunch of tasks

508
00:23:59,000 --> 00:24:01,780
to pick your head up
every once in a while

509
00:24:01,780 --> 00:24:04,970
and make sure that you're
focused on that feature.

510
00:24:04,970 --> 00:24:07,090
And potentially
you're asking, how

511
00:24:07,090 --> 00:24:11,120
does this task that I'm working
on fit into this feature?

512
00:24:11,120 --> 00:24:14,140
Because if you've got that
context, you may even realize,

513
00:24:14,140 --> 00:24:15,910
hey, my task
actually isn't going

514
00:24:15,910 --> 00:24:18,440
to help me deliver
on that feature.

515
00:24:18,440 --> 00:24:20,790
I'm not working on
the right thing.

516
00:24:20,790 --> 00:24:23,890
As an engineer with a
technical background,

517
00:24:23,890 --> 00:24:26,970
I'm building good software,
functional requirements,

518
00:24:26,970 --> 00:24:29,550
et cetera.

519
00:24:29,550 --> 00:24:33,220
You really need to be
focusing on that feature.

520
00:24:33,220 --> 00:24:34,940
Because your task
is just a means

521
00:24:34,940 --> 00:24:36,540
to delivering on that feature.

522
00:24:39,570 --> 00:24:44,230
So in summary, just a little
bit of philosophy type stuff.

523
00:24:44,230 --> 00:24:45,590
Embrace iteration.

524
00:24:45,590 --> 00:24:47,170
Challenge the impossible.

525
00:24:47,170 --> 00:24:50,840
Value proven code and focus
on features, not tasks.

526
00:24:50,840 --> 00:24:54,390
And that's kind of my top four.

527
00:24:54,390 --> 00:24:57,242
It's easily a top 10 list.

528
00:24:57,242 --> 00:24:58,700
The thing is to
kind of think about

529
00:24:58,700 --> 00:25:03,730
in terms of coming into a big
company, working on a big team.

530
00:25:03,730 --> 00:25:06,490
Some of the philosophy that you
kind of need to take with you

531
00:25:06,490 --> 00:25:08,470
in your approach on
integrating with that team

532
00:25:08,470 --> 00:25:09,344
and being successful.

533
00:25:13,100 --> 00:25:15,890
Now I want to talk a little
bit about more on the process

534
00:25:15,890 --> 00:25:19,020
side, engineering quality.

535
00:25:19,020 --> 00:25:22,790
How do we as a engineering
organization at the studio help

536
00:25:22,790 --> 00:25:26,780
make sure that we're
delivering a quality product.

537
00:25:26,780 --> 00:25:32,460
And game development's
notoriously not very formal.

538
00:25:32,460 --> 00:25:36,070
We are game
developers, after all.

539
00:25:36,070 --> 00:25:38,440
The challenge, though,
when you're not

540
00:25:38,440 --> 00:25:40,815
terribly formal in your
development methodologies

541
00:25:40,815 --> 00:25:45,430
and your development
practices is success

542
00:25:45,430 --> 00:25:53,320
starts to rely on individual
heroics or individual heroism.

543
00:25:53,320 --> 00:25:55,302
And that's a bit
of a danger zone

544
00:25:55,302 --> 00:25:57,260
when you start talking
about shipping a quality

545
00:25:57,260 --> 00:26:00,270
product that's big and on time.

546
00:26:00,270 --> 00:26:02,190
Because quality's
never an accident.

547
00:26:02,190 --> 00:26:03,930
I love this quote.

548
00:26:03,930 --> 00:26:04,970
It's never an accident.

549
00:26:04,970 --> 00:26:06,720
It's the result of
intelligent effort.

550
00:26:09,390 --> 00:26:12,660
Now, before we talk
about processes for this,

551
00:26:12,660 --> 00:26:14,910
let's decompose
quality a little bit.

552
00:26:14,910 --> 00:26:17,300
Because this kind of plays
into some of the thinking

553
00:26:17,300 --> 00:26:20,287
on our approach to quality.

554
00:26:20,287 --> 00:26:22,620
And really, it's going to
focus on the different aspects

555
00:26:22,620 --> 00:26:23,990
of quality.

556
00:26:23,990 --> 00:26:25,560
There's perceptual quality.

557
00:26:25,560 --> 00:26:26,750
Is the game fun?

558
00:26:26,750 --> 00:26:28,450
Is it beautiful?

559
00:26:28,450 --> 00:26:33,160
It's how the gamer
experiences the game.

560
00:26:33,160 --> 00:26:34,520
There's functional quality.

561
00:26:34,520 --> 00:26:36,760
That's kind of, does it work?

562
00:26:36,760 --> 00:26:40,740
An engineer is given
a spec and writes code

563
00:26:40,740 --> 00:26:44,890
and it either does or does not
do what it was intended to do.

564
00:26:44,890 --> 00:26:47,770
And then there's
structural quality.

565
00:26:47,770 --> 00:26:50,744
That's the underlying,
behind the scenes quality.

566
00:26:50,744 --> 00:26:52,160
It tends to be the
type of quality

567
00:26:52,160 --> 00:26:54,520
that engineers care
about the most.

568
00:26:54,520 --> 00:26:56,940
That's kind of the
code quality space.

569
00:26:56,940 --> 00:27:00,380
Is my code perfect?

570
00:27:00,380 --> 00:27:03,990
The top two perceptual quality,
functional quality, those

571
00:27:03,990 --> 00:27:07,330
are things that the gamer is
going to see in your product.

572
00:27:07,330 --> 00:27:09,480
Structural quality
is not something

573
00:27:09,480 --> 00:27:12,400
that the gamer will
actually ever see or notice.

574
00:27:12,400 --> 00:27:15,150
But it is something that
will hold up your project.

575
00:27:15,150 --> 00:27:17,801
If you don't have the right
level of structural quality

576
00:27:17,801 --> 00:27:19,300
in your product,
it will prevent you

577
00:27:19,300 --> 00:27:22,990
from delivering that product.

578
00:27:22,990 --> 00:27:25,060
Now ultimately, at
the end of the day

579
00:27:25,060 --> 00:27:29,480
we're probably most interested
in that perceptual quality.

580
00:27:29,480 --> 00:27:31,200
Because that's the
one that's most

581
00:27:31,200 --> 00:27:32,890
going to resonate
with the gamers.

582
00:27:32,890 --> 00:27:33,560
Is it fun?

583
00:27:33,560 --> 00:27:34,640
Is it beautiful?

584
00:27:34,640 --> 00:27:36,600
And we kind of
like it meta critic

585
00:27:36,600 --> 00:27:40,520
as one possible measure of
whether our game's being

586
00:27:40,520 --> 00:27:43,970
perceived as a high quality.

587
00:27:43,970 --> 00:27:46,960
This is basically a reviewer
sits down, plays the game,

588
00:27:46,960 --> 00:27:49,270
or even if you go
look at user reviews.

589
00:27:49,270 --> 00:27:51,550
A gamer sits down
and plays the game.

590
00:27:51,550 --> 00:27:54,130
And then that game
experience gets

591
00:27:54,130 --> 00:27:56,420
processed through
their filters and they

592
00:27:56,420 --> 00:28:01,590
perceive it to be at a
certain level of quality.

593
00:28:01,590 --> 00:28:03,160
Now, functional
quality is inevitably

594
00:28:03,160 --> 00:28:05,630
going to play into this.

595
00:28:05,630 --> 00:28:07,990
But we generally see, I
think, with meta critic

596
00:28:07,990 --> 00:28:11,350
is a lot of the
functional issues may not

597
00:28:11,350 --> 00:28:15,260
be seen because it takes hours
and hours and hours of playing

598
00:28:15,260 --> 00:28:17,860
the game before
those maybe surfaced.

599
00:28:17,860 --> 00:28:21,180
And it tends to be,
especially with meta critic,

600
00:28:21,180 --> 00:28:24,020
you spend four to eight
hours playing the game,

601
00:28:24,020 --> 00:28:28,300
you process that
experience, give it a score.

602
00:28:28,300 --> 00:28:30,490
So I tend to kind of
argue, at least when

603
00:28:30,490 --> 00:28:33,040
we're talking internally
about quality, only

604
00:28:33,040 --> 00:28:35,250
the absolute worst
functional issues are

605
00:28:35,250 --> 00:28:37,760
going to show up and play
into your meta critic score.

606
00:28:37,760 --> 00:28:41,142
But the gamer who plays
your game for 100 plus hours

607
00:28:41,142 --> 00:28:43,600
is really going to notice a
lot of those functional quality

608
00:28:43,600 --> 00:28:44,100
issues.

609
00:28:47,190 --> 00:28:49,170
We also have been
talking recently

610
00:28:49,170 --> 00:28:52,570
about kind of a
spectrum of quality.

611
00:28:52,570 --> 00:28:55,390
And this is kind of
tied to meta critic.

612
00:28:55,390 --> 00:28:58,750
And I steal this from one
of our executive producers

613
00:28:58,750 --> 00:29:04,520
where from 0 to about 69,
those games are sort of just

614
00:29:04,520 --> 00:29:07,250
foundationally solid.

615
00:29:07,250 --> 00:29:10,590
If you're anywhere below
69, sort of the upper end

616
00:29:10,590 --> 00:29:11,300
of that spectrum.

617
00:29:11,300 --> 00:29:15,460
You get to about 69 or 70 by
being foundationally solid.

618
00:29:15,460 --> 00:29:18,520
Anything below that, you're
buggy, you're broken,

619
00:29:18,520 --> 00:29:21,530
you've got a lot of issues.

620
00:29:21,530 --> 00:29:23,920
And from there,
quality kind of builds.

621
00:29:23,920 --> 00:29:26,997
If you're buggy, broken, and
have just glaring problems,

622
00:29:26,997 --> 00:29:28,830
you're never going to
get to that next level

623
00:29:28,830 --> 00:29:30,790
where you've got good function.

624
00:29:30,790 --> 00:29:33,560
It functions and if you have
a competitor in the space,

625
00:29:33,560 --> 00:29:36,300
it functions at least as
well as the competitor.

626
00:29:36,300 --> 00:29:40,587
And that's kind of
that 70 to 79 range.

627
00:29:40,587 --> 00:29:43,170
But what we really want to do
is we want to get up above that.

628
00:29:43,170 --> 00:29:45,170
This is interactive
entertainment.

629
00:29:45,170 --> 00:29:49,420
We're trying to really delight
the gamers that play our game.

630
00:29:49,420 --> 00:29:52,020
And from 80 to 89,
we kind of start

631
00:29:52,020 --> 00:29:54,020
to talk about, all right,
that's the point where

632
00:29:54,020 --> 00:29:55,400
your game is getting immersive.

633
00:29:59,670 --> 00:30:02,410
Especially for Madden, it's
starting to be very realistic

634
00:30:02,410 --> 00:30:05,640
and they feel like they're
there on Sunday watching a game

635
00:30:05,640 --> 00:30:08,620
and they're sort of
immersed in the experience.

636
00:30:08,620 --> 00:30:12,580
And then even above
that, which is

637
00:30:12,580 --> 00:30:15,830
where we really want to get
to is 90 to 99 or even 100.

638
00:30:15,830 --> 00:30:18,150
I don't know if it
actually goes to 100.

639
00:30:18,150 --> 00:30:20,020
But that's where
now not only are

640
00:30:20,020 --> 00:30:22,880
you immersed in the
experience, but the game

641
00:30:22,880 --> 00:30:23,735
is evoking emotion.

642
00:30:29,130 --> 00:30:32,300
So when you get up
into those upper bands,

643
00:30:32,300 --> 00:30:35,292
that's where perceptual quality
really starts to come in.

644
00:30:35,292 --> 00:30:36,750
I'm going to talk
a little bit more

645
00:30:36,750 --> 00:30:38,916
about that foundational and
that functional quality,

646
00:30:38,916 --> 00:30:43,470
because that's what engineers
and development directors are

647
00:30:43,470 --> 00:30:46,480
ultimately paid to solve.

648
00:30:46,480 --> 00:30:49,980
And so the question
is, on a big project

649
00:30:49,980 --> 00:30:56,845
with a lot of complexity that's
got to hit a particular date,

650
00:30:56,845 --> 00:30:58,470
inevitably need to
put a lot of process

651
00:30:58,470 --> 00:31:01,680
in place to try to do make
sure that you're putting out

652
00:31:01,680 --> 00:31:04,620
that quality product or
at least at a minimum

653
00:31:04,620 --> 00:31:06,760
were able to hit that
foundational and functional

654
00:31:06,760 --> 00:31:09,490
quality so that the
designers and everyone

655
00:31:09,490 --> 00:31:12,345
else can help push us up
into those perceptual realms.

656
00:31:14,870 --> 00:31:17,255
The first process
is code review.

657
00:31:17,255 --> 00:31:19,130
So let's talk a little
bit about code review.

658
00:31:19,130 --> 00:31:22,530
Because engineers
love code review.

659
00:31:22,530 --> 00:31:25,560
Especially because
their code is perfect

660
00:31:25,560 --> 00:31:27,637
and no one needs to
look at this code

661
00:31:27,637 --> 00:31:29,220
to figure out whether
or not it should

662
00:31:29,220 --> 00:31:30,428
be allowed to get checked in.

663
00:31:32,870 --> 00:31:35,020
At EA and on Madden
in particular,

664
00:31:35,020 --> 00:31:38,250
we do have a blanket policy,
all code gets reviewed.

665
00:31:38,250 --> 00:31:40,870
If you're going to check code
in, it's got to get reviewed.

666
00:31:40,870 --> 00:31:43,170
We even in this day
and age where data

667
00:31:43,170 --> 00:31:45,039
is a huge part of development.

668
00:31:45,039 --> 00:31:47,080
Actually there's a lot of
data that gets reviewed

669
00:31:47,080 --> 00:31:49,590
before it gets checked in.

670
00:31:49,590 --> 00:31:52,102
But code review is sort
of the primary function

671
00:31:52,102 --> 00:31:54,060
where all right, you're
going to check in code.

672
00:31:54,060 --> 00:31:55,101
It needs to get reviewed.

673
00:31:55,101 --> 00:31:57,955
And the goal there, let's
make sure there's no bugs.

674
00:32:00,870 --> 00:32:04,050
Get a second set of eyes,
have someone look at it.

675
00:32:04,050 --> 00:32:05,122
Shake out all the bugs.

676
00:32:05,122 --> 00:32:07,580
Because the second set of eyes
will find the bugs before it

677
00:32:07,580 --> 00:32:10,450
makes it to QA.

678
00:32:10,450 --> 00:32:14,610
There was an interesting
study recently, Rebel Labs.

679
00:32:14,610 --> 00:32:19,250
They did a survey and
asked development teams,

680
00:32:19,250 --> 00:32:21,180
how many of you do code
review and how often

681
00:32:21,180 --> 00:32:23,610
do you code review?

682
00:32:23,610 --> 00:32:26,160
Code review is very
prevalent coming out

683
00:32:26,160 --> 00:32:28,690
from this industry survey.

684
00:32:28,690 --> 00:32:33,745
I think it was 76% of teams
do a lot of code review.

685
00:32:33,745 --> 00:32:35,660
And their ultimate
goal with this survey

686
00:32:35,660 --> 00:32:40,690
was to figure out the impact
of code review on quality.

687
00:32:40,690 --> 00:32:44,720
And quality for them was
defined by lack of bugs.

688
00:32:44,720 --> 00:32:49,270
So the functional
realm of quality.

689
00:32:49,270 --> 00:32:52,687
And their study produced kind
of an interesting result.

690
00:32:52,687 --> 00:32:54,770
They were actually looking
at two factors, quality

691
00:32:54,770 --> 00:32:56,330
and predictability.

692
00:32:56,330 --> 00:32:58,992
Both super important for sports.

693
00:32:58,992 --> 00:33:01,070
And they found that
code review really

694
00:33:01,070 --> 00:33:02,720
helped with predictability.

695
00:33:02,720 --> 00:33:04,250
It indexed well.

696
00:33:04,250 --> 00:33:09,670
You could see if you did no code
review, there was an 11% or 10%

697
00:33:09,670 --> 00:33:14,030
swing between no code review
and always doing code review.

698
00:33:14,030 --> 00:33:18,140
Much less influence
though on quality.

699
00:33:18,140 --> 00:33:19,950
They found that there
was not actually

700
00:33:19,950 --> 00:33:23,330
not much of a difference between
teams that did code review

701
00:33:23,330 --> 00:33:26,080
and didn't do code review
in the span of quality.

702
00:33:26,080 --> 00:33:28,020
Again, quality is
measured as bugs.

703
00:33:28,020 --> 00:33:29,794
AUDIENCE: What is
predictability?

704
00:33:29,794 --> 00:33:32,210
TIMOTHY COWAN: Predictability
is able to hit your schedule

705
00:33:32,210 --> 00:33:32,709
on time.

706
00:33:37,590 --> 00:33:40,307
Now, that's not to say that we
don't find bugs in code review.

707
00:33:40,307 --> 00:33:42,640
But I think that's interesting
because it kind of aligns

708
00:33:42,640 --> 00:33:46,210
with some of my experience
within EA that, now, don't get

709
00:33:46,210 --> 00:33:48,450
me wrong, we put code
review in place because we

710
00:33:48,450 --> 00:33:50,560
wanted to find bugs early.

711
00:33:50,560 --> 00:33:53,690
And when I started at EA we
did not do any code review.

712
00:33:53,690 --> 00:33:56,810
In 2001, again, this is
kind of individual heroics,

713
00:33:56,810 --> 00:33:58,241
pretty informal.

714
00:33:58,241 --> 00:34:01,940
On a 200 person team, you've
got to do code review.

715
00:34:01,940 --> 00:34:03,619
And we do want to find bugs.

716
00:34:03,619 --> 00:34:05,160
But it's actually
a little surprising

717
00:34:05,160 --> 00:34:07,920
that we don't find many
bugs in code review.

718
00:34:07,920 --> 00:34:11,520
And I see that at
our studio as well.

719
00:34:11,520 --> 00:34:12,963
How many people here know C++?

720
00:34:15,510 --> 00:34:16,010
All right.

721
00:34:16,010 --> 00:34:17,380
Awesome.

722
00:34:17,380 --> 00:34:20,199
Let's play a little
game quick here.

723
00:34:20,199 --> 00:34:21,509
It's called find the bug.

724
00:34:21,509 --> 00:34:22,675
You guys are code reviewers.

725
00:34:26,060 --> 00:34:29,550
Raise your hand if
you can spot the bug.

726
00:34:29,550 --> 00:34:31,512
There is a bug here.

727
00:34:31,512 --> 00:34:32,428
AUDIENCE: [INAUDIBLE].

728
00:34:36,130 --> 00:34:38,510
TIMOTHY COWAN: You can't see it?

729
00:34:38,510 --> 00:34:39,010
OK.

730
00:34:43,170 --> 00:34:44,850
Is it the contrast
or the lights?

731
00:34:53,420 --> 00:34:55,579
Anybody that can see it spot it?

732
00:34:55,579 --> 00:34:56,495
AUDIENCE: [INAUDIBLE].

733
00:35:04,920 --> 00:35:06,420
TIMOTHY COWAN: Good.

734
00:35:06,420 --> 00:35:08,950
Good.

735
00:35:08,950 --> 00:35:10,590
That is the mistake.

736
00:35:10,590 --> 00:35:13,720
It's a cut and paste error.

737
00:35:13,720 --> 00:35:15,830
So someone moving fast.

738
00:35:15,830 --> 00:35:19,640
And it tends to be pretty
easy not to notice this.

739
00:35:19,640 --> 00:35:22,930
It's just kind of hidden
in a block of code.

740
00:35:22,930 --> 00:35:24,760
It actually was
intended in this case.

741
00:35:24,760 --> 00:35:26,970
It's just kind of following
a pretty simple pattern.

742
00:35:26,970 --> 00:35:32,890
They should have been copying
PURL into the STR web URL.

743
00:35:32,890 --> 00:35:38,360
This was code that was written,
tested by the engineer that

744
00:35:38,360 --> 00:35:41,190
wrote it, I presume.

745
00:35:41,190 --> 00:35:46,970
Code reviewed, checked in, and
made it all the way out to QA.

746
00:35:46,970 --> 00:35:49,280
I don't know if an
actual bug manifested,

747
00:35:49,280 --> 00:35:52,590
but this code did make
it through code review.

748
00:35:52,590 --> 00:35:54,570
In fact, all this code
we're going to look at

749
00:35:54,570 --> 00:35:57,840
made it through code review.

750
00:35:57,840 --> 00:35:59,576
How about this one?

751
00:35:59,576 --> 00:36:02,041
AUDIENCE: [INAUDIBLE].

752
00:36:02,041 --> 00:36:04,020
TIMOTHY COWAN: All right.

753
00:36:04,020 --> 00:36:04,990
Yeah.

754
00:36:04,990 --> 00:36:09,460
That is kind of the trick with
code is it's kind of tiny text.

755
00:36:09,460 --> 00:36:11,925
[LAUGHTER]

756
00:36:15,395 --> 00:36:16,520
It's probably the sunlight.

757
00:36:25,280 --> 00:36:27,510
People in the front
row can see it?

758
00:36:27,510 --> 00:36:28,010
No?

759
00:36:33,079 --> 00:36:34,620
I guess this game's
a little less fun

760
00:36:34,620 --> 00:36:38,030
than maybe I was hoping.

761
00:36:38,030 --> 00:36:40,220
In this case, you've
got two different files

762
00:36:40,220 --> 00:36:42,824
and this is a function call.

763
00:36:42,824 --> 00:36:44,240
And there's one
line of code where

764
00:36:44,240 --> 00:36:46,120
they're using the function.

765
00:36:46,120 --> 00:36:47,680
In the second file,
if you're doing

766
00:36:47,680 --> 00:36:49,890
a code review you've actually
got to-- in this case,

767
00:36:49,890 --> 00:36:51,920
someone added a line of code.

768
00:36:51,920 --> 00:36:53,340
And when you get
that code review,

769
00:36:53,340 --> 00:36:55,110
you get just that file
with a little diff

770
00:36:55,110 --> 00:36:57,646
and you see their line
of code that they added.

771
00:36:57,646 --> 00:36:59,020
You won't actually
spot this blog

772
00:36:59,020 --> 00:37:00,700
unless you open
up the other file

773
00:37:00,700 --> 00:37:03,290
that they didn't edit and
go look at that function

774
00:37:03,290 --> 00:37:05,430
and how it's used.

775
00:37:05,430 --> 00:37:08,890
Because they've actually got
the two parameters reversed.

776
00:37:08,890 --> 00:37:10,550
They're both bools.

777
00:37:10,550 --> 00:37:14,040
Track actives versus
track passives.

778
00:37:14,040 --> 00:37:17,270
And the function they're
calling has those two parameters

779
00:37:17,270 --> 00:37:19,130
reversed.

780
00:37:19,130 --> 00:37:25,800
Again, written, tested,
code reviewed, checked in.

781
00:37:25,800 --> 00:37:28,470
I don't know if this
one's readable at all.

782
00:37:28,470 --> 00:37:29,970
I kind of have to
skip through this.

783
00:37:39,352 --> 00:37:41,319
In the interest of time,
I'll go ahead and just

784
00:37:41,319 --> 00:37:42,110
skip through these.

785
00:37:42,110 --> 00:37:47,230
Because I think they're all
somewhat fuzzy and small text.

786
00:37:47,230 --> 00:37:52,530
In this case, if you don't know
C++ or don't know the syntax,

787
00:37:52,530 --> 00:37:55,260
with that indentation there's
an if statement there.

788
00:37:55,260 --> 00:37:57,690
With the indentation
it was probably

789
00:37:57,690 --> 00:37:59,210
expected that that
code would only

790
00:37:59,210 --> 00:38:02,930
get it executed if the
if condition was true.

791
00:38:02,930 --> 00:38:06,440
They missed their curly braces.

792
00:38:06,440 --> 00:38:08,180
And that that
indentation tends to,

793
00:38:08,180 --> 00:38:11,040
if you're reviewing the
code, kind of draw your eye.

794
00:38:11,040 --> 00:38:13,042
It's like, yeah,
that code looks OK.

795
00:38:13,042 --> 00:38:14,250
You're kind of inspecting it.

796
00:38:14,250 --> 00:38:16,680
And I'm sure that they
knew what they were doing.

797
00:38:16,680 --> 00:38:21,140
And you kind of missed that
the braces aren't there.

798
00:38:21,140 --> 00:38:22,800
And this one's maybe-- no.

799
00:38:27,700 --> 00:38:32,020
So in this case,
their if statement,

800
00:38:32,020 --> 00:38:33,430
it's actually a constant number.

801
00:38:33,430 --> 00:38:36,560
This will always be true.

802
00:38:36,560 --> 00:38:40,620
And they probably actually
meant to check if that value was

803
00:38:40,620 --> 00:38:43,250
equal to some particular value.

804
00:38:43,250 --> 00:38:46,510
But they actually just
put the mask right

805
00:38:46,510 --> 00:38:47,470
into the if statement.

806
00:38:47,470 --> 00:38:51,170
And that will always
be a number greater

807
00:38:51,170 --> 00:38:52,710
than 0, which will be true.

808
00:38:55,910 --> 00:38:57,860
And last one.

809
00:38:57,860 --> 00:38:59,310
This last statement here.

810
00:38:59,310 --> 00:39:02,290
This last statement
doesn't do anything.

811
00:39:02,290 --> 00:39:06,620
They're just assigning
one variable to itself.

812
00:39:06,620 --> 00:39:08,760
Again, really easy
cut and paste errors.

813
00:39:12,290 --> 00:39:15,042
And I think the surprising
thing in a lot of cases, again,

814
00:39:15,042 --> 00:39:16,000
we've got this process.

815
00:39:18,680 --> 00:39:20,579
We do find bugs.

816
00:39:20,579 --> 00:39:21,870
But you don't find all of them.

817
00:39:21,870 --> 00:39:25,790
And a lot a really simple ones,
things that after the fact,

818
00:39:25,790 --> 00:39:28,390
when you get a bug report from
QA and you go look at that

819
00:39:28,390 --> 00:39:32,000
and you're like, how
come no one caught this?

820
00:39:32,000 --> 00:39:34,480
How come the person that
wrote it didn't catch it?

821
00:39:34,480 --> 00:39:36,760
How come the code
reviewer didn't catch it?

822
00:39:36,760 --> 00:39:38,510
I'll tell you, in some
cases it's not even

823
00:39:38,510 --> 00:39:41,390
just one person code reviewing,
but you may have five people

824
00:39:41,390 --> 00:39:42,860
that code review it.

825
00:39:42,860 --> 00:39:45,270
Again, we're big,
multi-disciplinary.

826
00:39:45,270 --> 00:39:49,610
Sometimes one check in may take
five different subject matter

827
00:39:49,610 --> 00:39:51,360
experts where I need
a rendering engineer,

828
00:39:51,360 --> 00:39:53,610
I need a physics engineer,
and I need a UI engineer

829
00:39:53,610 --> 00:39:55,310
to review this code.

830
00:39:55,310 --> 00:39:58,710
And five different people review
the code and no one catches

831
00:39:58,710 --> 00:40:01,050
the problem.

832
00:40:01,050 --> 00:40:04,110
But there are other
reasons for code review.

833
00:40:04,110 --> 00:40:06,320
And this is one of the
things I'm pretty adamant on.

834
00:40:06,320 --> 00:40:09,420
Because when that report
comes out that says,

835
00:40:09,420 --> 00:40:11,492
code review doesn't
really find bugs.

836
00:40:11,492 --> 00:40:13,200
And in fact, if you
don't do code review,

837
00:40:13,200 --> 00:40:16,420
you're not all that bad off
in terms of finding bugs.

838
00:40:16,420 --> 00:40:18,281
I say, wait a minute.

839
00:40:18,281 --> 00:40:20,280
We're not going to stop
our code review process.

840
00:40:20,280 --> 00:40:22,910
Because there's other reasons
for doing code review.

841
00:40:22,910 --> 00:40:26,010
And finding bugs is actually,
I would call, my fourth reason.

842
00:40:28,710 --> 00:40:31,890
Really, code review's about
monitoring structural quality.

843
00:40:31,890 --> 00:40:35,570
And that's why that report
says if you do code review,

844
00:40:35,570 --> 00:40:37,584
you get better predictability.

845
00:40:37,584 --> 00:40:39,000
Because your
structural quality is

846
00:40:39,000 --> 00:40:42,450
this underlying quality
that's going to help you

847
00:40:42,450 --> 00:40:43,310
be successful.

848
00:40:43,310 --> 00:40:45,310
It's going to help with
your predictability.

849
00:40:45,310 --> 00:40:47,840
It's going to help with
knock-on effects, where

850
00:40:47,840 --> 00:40:52,230
if you write code that's
consistent, reliable, robust,

851
00:40:52,230 --> 00:40:55,080
and file certain
programming patterns,

852
00:40:55,080 --> 00:40:57,880
you're going to end up with a
better result in the long term

853
00:40:57,880 --> 00:40:59,171
and you'll be more predictable.

854
00:41:01,370 --> 00:41:03,120
It's also about
knowledge sharing.

855
00:41:03,120 --> 00:41:06,080
So on a big team
of 150 people where

856
00:41:06,080 --> 00:41:07,680
everyone's got their
own little task

857
00:41:07,680 --> 00:41:10,590
and they own their
little task, we

858
00:41:10,590 --> 00:41:12,560
need to make sure that
the whole team has

859
00:41:12,560 --> 00:41:16,460
some insight into what all the
other developments are doing.

860
00:41:16,460 --> 00:41:20,330
And if you're out sick
or you win the lottery

861
00:41:20,330 --> 00:41:25,050
and stop working, someone else
at the company's has maybe

862
00:41:25,050 --> 00:41:27,260
done some code reviews and
has seen what you're doing

863
00:41:27,260 --> 00:41:31,020
and has learned from that and
knows about how that feature,

864
00:41:31,020 --> 00:41:33,440
how that task works.

865
00:41:33,440 --> 00:41:35,830
And then it's also
about education.

866
00:41:35,830 --> 00:41:40,120
So if you are the best
engineer on the planet,

867
00:41:40,120 --> 00:41:42,210
you've been writing
code for 14 years,

868
00:41:42,210 --> 00:41:44,460
and you've learned
from all your mistakes,

869
00:41:44,460 --> 00:41:49,891
and you write the most perfect
code, having a new engineer who

870
00:41:49,891 --> 00:41:51,265
just came to work
at the studios,

871
00:41:51,265 --> 00:41:52,780
is writing their
first line of code

872
00:41:52,780 --> 00:41:57,900
ever review your code teaches
that person something.

873
00:41:57,900 --> 00:41:59,936
And then the last point
is obviously catch bugs.

874
00:41:59,936 --> 00:42:01,810
We do want to catch bugs
through code review.

875
00:42:01,810 --> 00:42:04,210
It's just it's not the
only thing that you get out

876
00:42:04,210 --> 00:42:06,160
of code review.

877
00:42:06,160 --> 00:42:09,570
And really, static analysis
and rigorous testing

878
00:42:09,570 --> 00:42:11,460
is the best way to catch bugs.

879
00:42:11,460 --> 00:42:14,690
And in fact, all those
examples I just went through

880
00:42:14,690 --> 00:42:17,610
are caught by static
analysis that we apply

881
00:42:17,610 --> 00:42:19,690
to our code base every day.

882
00:42:19,690 --> 00:42:22,390
Static analysis today is
actually very good at pattern

883
00:42:22,390 --> 00:42:25,890
matching and identifying
those types of issues.

884
00:42:25,890 --> 00:42:29,400
And on a code base our
size, you see a lot of them.

885
00:42:29,400 --> 00:42:32,420
And static analysis is
actually a very important

886
00:42:32,420 --> 00:42:35,170
additional process, I'm not
going to talk a lot about that,

887
00:42:35,170 --> 00:42:37,925
to layer into a robust
development process.

888
00:42:41,420 --> 00:42:43,124
All right, so the next piece.

889
00:42:43,124 --> 00:42:44,540
All right, if we
want to find bugs

890
00:42:44,540 --> 00:42:46,040
we want to drive
functional quality,

891
00:42:46,040 --> 00:42:48,800
we need quality testing.

892
00:42:48,800 --> 00:42:50,550
And your game
development's messy.

893
00:42:50,550 --> 00:42:52,580
You're never going
to find all the bugs.

894
00:42:52,580 --> 00:42:56,194
And really testing is
only good to find bugs.

895
00:42:56,194 --> 00:42:57,610
No matter how much
testing you do,

896
00:42:57,610 --> 00:42:58,820
you're never going
to know what bugs

897
00:42:58,820 --> 00:43:00,210
you haven't found, obviously.

898
00:43:04,000 --> 00:43:07,420
So we've got to make
sure we do testing.

899
00:43:07,420 --> 00:43:09,980
And there's a lot of
different types of testing.

900
00:43:09,980 --> 00:43:12,890
There's the engineer doing
their own software testing.

901
00:43:12,890 --> 00:43:16,610
There's our QA department
who does black box testing.

902
00:43:16,610 --> 00:43:17,750
There's automated testing.

903
00:43:17,750 --> 00:43:19,240
There's a whole
spectrum of testing

904
00:43:19,240 --> 00:43:21,060
that we do on our
product to help

905
00:43:21,060 --> 00:43:24,180
make sure that the product
we put out is amazing.

906
00:43:24,180 --> 00:43:27,810
But inevitably, game
development's messy.

907
00:43:27,810 --> 00:43:31,280
No matter what we do,
things do slip out.

908
00:43:31,280 --> 00:43:35,830
In fact, in the first 48
hours of Madden launching,

909
00:43:35,830 --> 00:43:41,230
there is more testing done
on our product by the gamers

910
00:43:41,230 --> 00:43:44,750
than we do an entire 10 month
development cycle by ourselves.

911
00:43:48,300 --> 00:43:51,210
So one of the big processes
we have in place today

912
00:43:51,210 --> 00:43:53,670
is test automation.

913
00:43:53,670 --> 00:43:58,100
You can't hire enough QA
people to test your game

914
00:43:58,100 --> 00:44:01,530
for the scale and the size of
games and complexity of games

915
00:44:01,530 --> 00:44:02,310
today.

916
00:44:02,310 --> 00:44:04,290
So we do lots and lots
of test automation.

917
00:44:04,290 --> 00:44:07,030
And basically the goal is to
get hours and hours and hours

918
00:44:07,030 --> 00:44:10,600
of testing through automation
on hundreds of dev kits

919
00:44:10,600 --> 00:44:15,510
that sit idle at the studio at
night when people are at home.

920
00:44:15,510 --> 00:44:16,962
There's one big
piece that I want

921
00:44:16,962 --> 00:44:18,670
to talk about with
test automation, which

922
00:44:18,670 --> 00:44:21,565
is when we talk to
engineering, verification,

923
00:44:21,565 --> 00:44:25,060
and validation,
testing is a means

924
00:44:25,060 --> 00:44:29,660
to compare the execution,
the results of the test

925
00:44:29,660 --> 00:44:33,420
against the expected
results for that test.

926
00:44:33,420 --> 00:44:36,510
And one thing that we're
starting to do a lot more at EA

927
00:44:36,510 --> 00:44:39,040
is we're kind of
looking at testing where

928
00:44:39,040 --> 00:44:41,740
automated testing in particular
where kind of the only expected

929
00:44:41,740 --> 00:44:44,950
result is that the
test completes.

930
00:44:44,950 --> 00:44:47,250
That test may take five minutes.

931
00:44:47,250 --> 00:44:50,250
It may take 50 minutes.

932
00:44:50,250 --> 00:44:53,670
All we're doing is looking,
hey, did that test complete.

933
00:44:53,670 --> 00:44:56,180
And within that span
of say 50 minutes,

934
00:44:56,180 --> 00:44:58,830
there's all kinds of
other things that went on.

935
00:44:58,830 --> 00:45:01,600
And we're not really testing
whether all those things

936
00:45:01,600 --> 00:45:07,530
between here and here met
what our expected results are.

937
00:45:07,530 --> 00:45:11,600
So really, this is about
measurement and functional

938
00:45:11,600 --> 00:45:17,010
quality measurement, which
starts with technical KPIs.

939
00:45:17,010 --> 00:45:21,020
Over a 50 minute test, did
the game run at frame rate?

940
00:45:21,020 --> 00:45:23,670
What are the load times
between your different modes?

941
00:45:23,670 --> 00:45:26,780
Did they meet what our
expected requirements were?

942
00:45:26,780 --> 00:45:32,140
Obviously there's crashes,
desyncs, and disconnects,

943
00:45:32,140 --> 00:45:35,920
which are very abrupt
failures for that test where

944
00:45:35,920 --> 00:45:37,300
it won't actually complete.

945
00:45:37,300 --> 00:45:39,520
But there's all these
other technical KPIs

946
00:45:39,520 --> 00:45:41,620
where the test will
continue, but it may not

947
00:45:41,620 --> 00:45:45,614
have been doing what
you'd expect it to do.

948
00:45:45,614 --> 00:45:46,530
AUDIENCE: [INAUDIBLE].

949
00:45:54,750 --> 00:45:56,940
TIMOTHY COWAN: So what we're
starting to do more now

950
00:45:56,940 --> 00:46:01,990
is measure and make sure that
every build so that every test

951
00:46:01,990 --> 00:46:03,380
where an automated
test, no one's

952
00:46:03,380 --> 00:46:05,230
sitting there watching it.

953
00:46:05,230 --> 00:46:06,280
I may not know.

954
00:46:06,280 --> 00:46:09,510
If a QA person doesn't sit there
and watch that automated test,

955
00:46:09,510 --> 00:46:12,070
we don't know that it's not
running at target frame rate

956
00:46:12,070 --> 00:46:14,875
unless we measure
it and report it.

957
00:46:14,875 --> 00:46:16,920
And with online
services today, you

958
00:46:16,920 --> 00:46:20,020
can do a lot of measuring
and automated reporting,

959
00:46:20,020 --> 00:46:23,460
capture all that data,
and create dashboards

960
00:46:23,460 --> 00:46:26,650
that can show, all right, what's
the performance of my Game

961
00:46:26,650 --> 00:46:29,310
this is one the first ones we
put in over the last couple

962
00:46:29,310 --> 00:46:29,810
years.

963
00:46:29,810 --> 00:46:32,130
We've always had
performance analysis.

964
00:46:32,130 --> 00:46:35,300
But what we now have
is automated solutions

965
00:46:35,300 --> 00:46:38,170
with very detailed
level of reports,

966
00:46:38,170 --> 00:46:41,580
so that every time
someone plays our game,

967
00:46:41,580 --> 00:46:44,260
we're getting frame rate
reports out of that game.

968
00:46:46,940 --> 00:46:49,600
And you can drill down a little
bit to look at for Madden

969
00:46:49,600 --> 00:46:51,250
by stadium.

970
00:46:51,250 --> 00:46:53,680
Every game that's been
played in each stadium, what

971
00:46:53,680 --> 00:46:54,930
does the frame rate look like?

972
00:46:54,930 --> 00:46:57,870
And look and see are
there certain stadiums

973
00:46:57,870 --> 00:47:00,530
that have problems.

974
00:47:00,530 --> 00:47:03,607
And you can even look
at an individual session

975
00:47:03,607 --> 00:47:05,940
and say, all right, was there
an individual game session

976
00:47:05,940 --> 00:47:10,710
that had a strange
frame rate problem?

977
00:47:10,710 --> 00:47:13,800
Now beyond our technical
KPIs, those key performance

978
00:47:13,800 --> 00:47:17,310
indicators, there's other
types of functional issues

979
00:47:17,310 --> 00:47:20,030
that we need to look to measure.

980
00:47:20,030 --> 00:47:23,430
Bugs do slip out into the wild.

981
00:47:23,430 --> 00:47:25,880
This was one that got a
lot of publicity this year,

982
00:47:25,880 --> 00:47:26,585
the tiny titan.

983
00:47:30,160 --> 00:47:33,410
UFC had some well known
glitches in its game

984
00:47:33,410 --> 00:47:38,139
where a lot of funny animation
and physics problems.

985
00:47:38,139 --> 00:47:39,680
Inevitably when
these types of things

986
00:47:39,680 --> 00:47:43,060
happen, if you're familiar with
the notion of asserts, sort

987
00:47:43,060 --> 00:47:46,300
of non fatal errors or
warnings that engineers put

988
00:47:46,300 --> 00:47:50,510
into their code to catch non
fatal problems and warn them

989
00:47:50,510 --> 00:47:52,870
that, hey, you might
want to look at this.

990
00:47:52,870 --> 00:47:56,370
I guarantee when both of those
problems showed up in a debug

991
00:47:56,370 --> 00:47:59,190
build, an assert probably
would've yelled at a developer

992
00:47:59,190 --> 00:48:02,490
and m, hey, something's wrong.

993
00:48:02,490 --> 00:48:06,410
So there's this class of non
fatal functional problems

994
00:48:06,410 --> 00:48:09,080
that are pretty severe, because
if they get out into the wild,

995
00:48:09,080 --> 00:48:13,912
this breaks the immersion and
ability to experience emotion

996
00:48:13,912 --> 00:48:14,870
when you play the game.

997
00:48:14,870 --> 00:48:16,800
Or at least in some
cases, this one's

998
00:48:16,800 --> 00:48:18,040
probably a negative emotion.

999
00:48:21,230 --> 00:48:23,630
We need to start to
shift our focus on as you

1000
00:48:23,630 --> 00:48:25,160
build engineering systems.

1001
00:48:25,160 --> 00:48:29,720
This is, again, as an engineer,
what you bring to the table

1002
00:48:29,720 --> 00:48:34,450
is this methodology and process
of verifying and validating

1003
00:48:34,450 --> 00:48:36,289
against functional requirements.

1004
00:48:36,289 --> 00:48:38,080
Make sure as you're
building your features,

1005
00:48:38,080 --> 00:48:39,746
you're thinking about
testing and you're

1006
00:48:39,746 --> 00:48:41,270
thinking about measuring.

1007
00:48:41,270 --> 00:48:43,510
Is this doing what
it's supposed to do?

1008
00:48:43,510 --> 00:48:48,400
Because I've got a code base
of 10 million lines of code,

1009
00:48:48,400 --> 00:48:50,700
200 people contributing to it.

1010
00:48:50,700 --> 00:48:52,450
There's a lot of complexity.

1011
00:48:52,450 --> 00:48:55,510
And you never know throughout
that software development life

1012
00:48:55,510 --> 00:48:57,800
cycle what types of
problems might show up.

1013
00:48:57,800 --> 00:49:00,470
So as you're going into your
design and requirements,

1014
00:49:00,470 --> 00:49:02,990
start thinking about what
types of failure modes

1015
00:49:02,990 --> 00:49:06,730
are critical that I should
be measuring and reporting

1016
00:49:06,730 --> 00:49:09,540
through online systems.

1017
00:49:09,540 --> 00:49:14,650
We're starting to do a lot
more of this at our studio.

1018
00:49:14,650 --> 00:49:18,860
So in summary, we really talked
about two things, code review

1019
00:49:18,860 --> 00:49:20,230
and testing.

1020
00:49:20,230 --> 00:49:24,130
And specifically measurement
when you're doing that testing.

1021
00:49:24,130 --> 00:49:27,600
But it's really about,
for me, an engineer's

1022
00:49:27,600 --> 00:49:29,300
responsible for
driving the functional

1023
00:49:29,300 --> 00:49:31,760
quality of a product.

1024
00:49:31,760 --> 00:49:34,770
We write the code, we
write the specifications.

1025
00:49:34,770 --> 00:49:39,100
We have to drive the functional
quality of game software.

1026
00:49:39,100 --> 00:49:41,640
And a well executed project
has both the technology

1027
00:49:41,640 --> 00:49:45,150
and processes in place
to do that and make sure

1028
00:49:45,150 --> 00:49:47,690
that you're able to
take that big project

1029
00:49:47,690 --> 00:49:50,320
and get it done by
August every single year.

1030
00:49:52,890 --> 00:49:53,870
So that's it.

1031
00:49:53,870 --> 00:49:56,590
I think I maybe talked a little
bit longer than I originally

1032
00:49:56,590 --> 00:49:57,100
expected.

1033
00:49:57,100 --> 00:49:58,730
But we've got a
little bit of time

1034
00:49:58,730 --> 00:50:02,820
for Q&A. I think I was told
you guys take a break at 2:00.

1035
00:50:02,820 --> 00:50:05,810
But you can go ahead
and ask any questions

1036
00:50:05,810 --> 00:50:08,460
and I'll answer anything I can.

1037
00:50:08,460 --> 00:50:11,340
When we take the
break, if one on one

1038
00:50:11,340 --> 00:50:13,230
if you want to ask
more questions,

1039
00:50:13,230 --> 00:50:15,960
I'll probably hang around
while you're doing play testing

1040
00:50:15,960 --> 00:50:19,332
and can talk more.

1041
00:50:19,332 --> 00:50:19,915
Any questions?

1042
00:50:22,194 --> 00:50:23,110
AUDIENCE: [INAUDIBLE].

1043
00:50:27,040 --> 00:50:29,640
TIMOTHY COWAN:
Today it is all C++.

1044
00:50:29,640 --> 00:50:30,140
100%.

1045
00:50:33,440 --> 00:50:36,490
I did a talk yesterday sort
of to a general audience

1046
00:50:36,490 --> 00:50:38,650
about how the
industry's evolved.

1047
00:50:38,650 --> 00:50:41,850
There used to be a lot of
assembly language intermixed

1048
00:50:41,850 --> 00:50:43,125
with C++.

1049
00:50:43,125 --> 00:50:46,700
We don't do that anymore
today on today's consoles.

1050
00:50:46,700 --> 00:50:53,250
And there is in data
we do do Lua scripting.

1051
00:50:53,250 --> 00:50:55,920
There is ActionScript.

1052
00:50:55,920 --> 00:51:00,190
There is shader code and shaders
for graphics programming.

1053
00:51:00,190 --> 00:51:02,660
I bucket those more as data.

1054
00:51:02,660 --> 00:51:05,620
And they tend to be-- we have a
Lua interpreter that's written

1055
00:51:05,620 --> 00:51:10,172
in C++ the can load Lua
data scripts and process it.

1056
00:51:10,172 --> 00:51:11,630
We have an ActionScript
interpreter

1057
00:51:11,630 --> 00:51:14,360
that loads ActionScript
and processes it.

1058
00:51:14,360 --> 00:51:17,240
So generally when I'm talking
the 10 million lines of code,

1059
00:51:17,240 --> 00:51:20,000
it's the C++ game engine client.

1060
00:51:20,000 --> 00:51:22,230
But it does interact
with other types of code

1061
00:51:22,230 --> 00:51:23,744
that we consider data today.

1062
00:51:26,588 --> 00:51:30,380
AUDIENCE: So I've been in code
review situation all the way

1063
00:51:30,380 --> 00:51:33,080
from [INAUDIBLE]
mostly automated

1064
00:51:33,080 --> 00:51:35,348
through some sort of
code tracking system

1065
00:51:35,348 --> 00:51:39,046
or commit system where someone's
about to commit something,

1066
00:51:39,046 --> 00:51:43,090
[INAUDIBLE], or all the
way to the other end

1067
00:51:43,090 --> 00:51:46,430
where someone just grabs me and
makes me look at their screen.

1068
00:51:46,430 --> 00:51:48,330
How does that work?

1069
00:51:48,330 --> 00:51:51,640
Can you walk us through
how review works?

1070
00:51:51,640 --> 00:51:52,680
TIMOTHY COWAN: Sure.

1071
00:51:52,680 --> 00:51:56,145
The policy is all
code gets reviewed.

1072
00:51:58,806 --> 00:52:00,180
With that kind of
policy, there's

1073
00:52:00,180 --> 00:52:02,780
a lot of flexibility in
how the teams actually

1074
00:52:02,780 --> 00:52:04,710
implement the process.

1075
00:52:04,710 --> 00:52:10,020
But the tool set we give
them, we do have a server.

1076
00:52:10,020 --> 00:52:15,970
We actually use a software from
SmartBear called Collaborator.

1077
00:52:15,970 --> 00:52:18,080
And when you're going to
submit code to Perforce,

1078
00:52:18,080 --> 00:52:20,610
we use Perforce
for source control,

1079
00:52:20,610 --> 00:52:25,060
you upload the diffs to
Collaborator and assign

1080
00:52:25,060 --> 00:52:28,340
that to a reviewer to
go look at those diffs.

1081
00:52:28,340 --> 00:52:31,420
And they can mark it up, put
little bugs and comments,

1082
00:52:31,420 --> 00:52:34,060
send it back to you, and you
can kind of go back and forth.

1083
00:52:34,060 --> 00:52:35,160
So it's one option.

1084
00:52:35,160 --> 00:52:39,610
That's how 90% of the
code review gets done.

1085
00:52:39,610 --> 00:52:41,390
But the thing I tell
the teams is you

1086
00:52:41,390 --> 00:52:42,720
don't have to do it that way.

1087
00:52:42,720 --> 00:52:45,496
In fact, having someone come
to your desk and talk to you

1088
00:52:45,496 --> 00:52:47,120
and look at the code
over your shoulder

1089
00:52:47,120 --> 00:52:51,290
can, in a lot of cases,
be much higher value.

1090
00:52:51,290 --> 00:52:53,270
Because that can
help set some context

1091
00:52:53,270 --> 00:52:55,460
on why did you
write it this way,

1092
00:52:55,460 --> 00:52:57,300
and you can have a conversation.

1093
00:52:57,300 --> 00:53:00,220
So that's an option as well.

1094
00:53:00,220 --> 00:53:02,290
And you also have
the flexibility,

1095
00:53:02,290 --> 00:53:04,340
depending on the
situation, sometimes

1096
00:53:04,340 --> 00:53:07,110
code gets reviewed
after check in.

1097
00:53:07,110 --> 00:53:09,740
It doesn't have to get
reviewed prior to check in.

1098
00:53:09,740 --> 00:53:13,350
That's kind of up to the teams,
up to the urgency of the check

1099
00:53:13,350 --> 00:53:16,050
in, the seniority of the review.

1100
00:53:16,050 --> 00:53:19,160
But the request is that
all code gets reviewed.

1101
00:53:19,160 --> 00:53:22,720
Again, it kind of hit the
four bullet points on why.

1102
00:53:22,720 --> 00:53:25,180
So I guess basically it depends.

1103
00:53:27,856 --> 00:53:29,230
It is game
development, so we try

1104
00:53:29,230 --> 00:53:32,035
to be a little informal about
it and give you the flexibility

1105
00:53:32,035 --> 00:53:36,740
to do the review
the way you see fit.

1106
00:53:36,740 --> 00:53:40,450
But provide a lot of tools
and process and formality

1107
00:53:40,450 --> 00:53:45,570
kind of through a prescribed
automated system as well.

1108
00:53:45,570 --> 00:53:47,763
I think there was a
question in the back.

1109
00:53:47,763 --> 00:53:48,388
AUDIENCE: Yeah.

1110
00:53:48,388 --> 00:53:50,858
How much additional
work is for you

1111
00:53:50,858 --> 00:53:56,243
to do things like hoarding
the same game to Xbox,

1112
00:53:56,243 --> 00:53:57,225
PS1, [INAUDIBLE].

1113
00:54:00,820 --> 00:54:03,830
TIMOTHY COWAN: For today's
consoles, Xbox One,

1114
00:54:03,830 --> 00:54:05,630
PlayStation 4.

1115
00:54:05,630 --> 00:54:09,410
You're really starting to get
to hardware that's converging.

1116
00:54:09,410 --> 00:54:12,170
It's becoming more PC like.

1117
00:54:12,170 --> 00:54:14,300
Probably the biggest
difference is actually

1118
00:54:14,300 --> 00:54:16,900
going to be in the
operating system.

1119
00:54:16,900 --> 00:54:18,920
Each of those consoles
has an operating system.

1120
00:54:18,920 --> 00:54:23,050
Xbox One ends up being
sort of flavor of Windows.

1121
00:54:23,050 --> 00:54:28,470
And PlayStation 4 ends up
being some variety of Linux.

1122
00:54:28,470 --> 00:54:31,270
So you've got this
operating system and driver

1123
00:54:31,270 --> 00:54:34,100
layer that is different.

1124
00:54:34,100 --> 00:54:35,580
But with the way
EA's structured,

1125
00:54:35,580 --> 00:54:38,570
obviously we're cross platform.

1126
00:54:38,570 --> 00:54:41,190
Madden, when I showed
the charts earlier,

1127
00:54:41,190 --> 00:54:43,330
actually shipped on
seven different platforms

1128
00:54:43,330 --> 00:54:45,770
that one year for Madden 13.

1129
00:54:45,770 --> 00:54:50,410
We've always built
cross platform support.

1130
00:54:50,410 --> 00:54:55,710
I characterize it as about
a 70-30 to 80-20 rule, where

1131
00:54:55,710 --> 00:54:59,200
the majority of your
code has no idea what

1132
00:54:59,200 --> 00:55:01,690
hardware it's running on.

1133
00:55:01,690 --> 00:55:04,130
And there's just kind
of this base layer

1134
00:55:04,130 --> 00:55:05,610
that interacts
with the operating

1135
00:55:05,610 --> 00:55:10,000
system and the drivers and does
what it has to do specifically

1136
00:55:10,000 --> 00:55:11,790
for the console.

1137
00:55:11,790 --> 00:55:14,900
And so it's call it
80-20 in terms of code

1138
00:55:14,900 --> 00:55:18,640
and 80-20 in terms of effort.

1139
00:55:18,640 --> 00:55:24,110
So that 20% layer generally gets
built by our central technology

1140
00:55:24,110 --> 00:55:29,200
groups, whether it's the base
level rendering layer, the base

1141
00:55:29,200 --> 00:55:32,900
level file system layer, the
base level of memory management

1142
00:55:32,900 --> 00:55:34,670
layer.

1143
00:55:34,670 --> 00:55:36,690
At EA we've got
central technology

1144
00:55:36,690 --> 00:55:40,960
where that gets built
centrally and all teams use it.

1145
00:55:40,960 --> 00:55:42,700
And then the teams
take that technology

1146
00:55:42,700 --> 00:55:45,910
and build their platform
agnostic game on top of it

1147
00:55:45,910 --> 00:55:48,200
where it kind of doesn't
care whether it's

1148
00:55:48,200 --> 00:55:51,570
Xbox One or PlayStation 4.

1149
00:55:51,570 --> 00:55:53,320
But then the other
space that is different

1150
00:55:53,320 --> 00:55:58,350
is just really, I'd
say, performance.

1151
00:55:58,350 --> 00:56:00,620
Each of the systems has
slightly different bottlenecks

1152
00:56:00,620 --> 00:56:02,230
in terms of what's
fast and what's

1153
00:56:02,230 --> 00:56:08,141
not, whether it's memory
bandwidth, shader pipelines.

1154
00:56:08,141 --> 00:56:10,390
They have slightly different
strengths and weaknesses.

1155
00:56:10,390 --> 00:56:12,010
They're tuned differently.

1156
00:56:12,010 --> 00:56:16,010
So you tend to have to look
at your content and your data

1157
00:56:16,010 --> 00:56:18,700
and structure it
in a way where it's

1158
00:56:18,700 --> 00:56:21,330
optimal for the two systems.

1159
00:56:21,330 --> 00:56:25,920
In the space of just kind of
art contact, how many polys,

1160
00:56:25,920 --> 00:56:28,170
how your textures are
structured and bundled,

1161
00:56:28,170 --> 00:56:32,830
how you write your
shaders especially.

1162
00:56:32,830 --> 00:56:35,730
There's quite a bit of space,
quite a bit of, I'd say,

1163
00:56:35,730 --> 00:56:37,471
platform specific
optimization work

1164
00:56:37,471 --> 00:56:39,220
to make sure we're
really taking advantage

1165
00:56:39,220 --> 00:56:41,011
of the different
bottlenecks in the system.

1166
00:56:44,940 --> 00:56:46,440
AUDIENCE: Why did
you choose to join

1167
00:56:46,440 --> 00:56:47,731
[INAUDIBLE] in the first place?

1168
00:56:51,190 --> 00:56:54,490
TIMOTHY COWAN: To be
honest, I had no idea

1169
00:56:54,490 --> 00:56:57,510
that you could make a
career in video games.

1170
00:56:57,510 --> 00:57:01,060
Like I said in the
intro, I wasn't

1171
00:57:01,060 --> 00:57:02,627
looking to go work
on video games,

1172
00:57:02,627 --> 00:57:04,460
mostly because I didn't
know that you could.

1173
00:57:09,230 --> 00:57:11,760
One of the games I played
the most growing up actually

1174
00:57:11,760 --> 00:57:13,080
was Madden.

1175
00:57:13,080 --> 00:57:14,570
In fact, really football games.

1176
00:57:14,570 --> 00:57:17,760
Anywhere from Tecmo Bowl on the
original Nintendo Entertainment

1177
00:57:17,760 --> 00:57:22,430
System to Madden on
the Sega Genesis.

1178
00:57:22,430 --> 00:57:26,430
I played a lot of NFL
football games growing up.

1179
00:57:30,520 --> 00:57:32,650
It was actually the
NFL football games

1180
00:57:32,650 --> 00:57:35,000
that I played that got me
interested in the sport

1181
00:57:35,000 --> 00:57:37,630
of football versus the
other way around where

1182
00:57:37,630 --> 00:57:40,410
I was a fan of the sport
so I played the games.

1183
00:57:40,410 --> 00:57:44,810
It was actually the games as
a junior high, high school

1184
00:57:44,810 --> 00:57:47,800
student that I really enjoyed
playing those football

1185
00:57:47,800 --> 00:57:50,820
games the most and enjoyed
beating my friends at them

1186
00:57:50,820 --> 00:57:52,810
if I could.

1187
00:57:52,810 --> 00:57:56,080
And so when I got contacted
by the recruiter where

1188
00:57:56,080 --> 00:57:58,240
I'm going to go
work at Lockheed,

1189
00:57:58,240 --> 00:58:00,410
hey, come work on
the next generation

1190
00:58:00,410 --> 00:58:03,395
of interactive entertainment,
make a football simulator,

1191
00:58:03,395 --> 00:58:06,220
I was like, well,
that would be great.

1192
00:58:06,220 --> 00:58:09,270
I used to love
playing those games

1193
00:58:09,270 --> 00:58:13,300
and was really excited about
coming down and getting

1194
00:58:13,300 --> 00:58:15,235
to work on some of
the games that I loved

1195
00:58:15,235 --> 00:58:16,360
playing when I was younger.

1196
00:58:22,130 --> 00:58:22,790
Anything else?

1197
00:58:26,427 --> 00:58:29,760
AUDIENCE: Do know of SB Nation's
"Breaking Madden" series?

1198
00:58:29,760 --> 00:58:33,720
And if so, do you have
any opinions on that?

1199
00:58:33,720 --> 00:58:38,323
They deliberately set the game
up with extreme characters,

1200
00:58:38,323 --> 00:58:41,221
like 90% skill everything
and 0% skill everything.

1201
00:58:44,000 --> 00:58:45,250
TIMOTHY COWAN: I did actually.

1202
00:58:45,250 --> 00:58:48,350
I think it was probably about
two weeks ago I watched that.

1203
00:58:48,350 --> 00:58:50,362
So I do know what
you're talking about.

1204
00:58:50,362 --> 00:58:51,820
My opinion, I think
that's awesome.

1205
00:58:55,000 --> 00:58:58,100
Madden and EA
Sports, at its core,

1206
00:58:58,100 --> 00:59:00,330
is all about
authenticity and realism.

1207
00:59:00,330 --> 00:59:02,710
EA Sports, it's in the game.

1208
00:59:02,710 --> 00:59:05,890
And we're pretty serious about
that authenticity, realism,

1209
00:59:05,890 --> 00:59:08,790
almost to the point
that, it's just

1210
00:59:08,790 --> 00:59:10,930
me talking as a
gamer, where we tend

1211
00:59:10,930 --> 00:59:13,550
to be a little too serious.

1212
00:59:13,550 --> 00:59:15,390
And I kind of like
that kind of stuff

1213
00:59:15,390 --> 00:59:17,680
where it's more just
having fun with the game,

1214
00:59:17,680 --> 00:59:19,780
which is what it's all about.

1215
00:59:19,780 --> 00:59:24,670
And giving, I think, the
content creation tools

1216
00:59:24,670 --> 00:59:26,840
in the hands of the
gamers, where you're

1217
00:59:26,840 --> 00:59:28,880
starting to see
a lot more gamers

1218
00:59:28,880 --> 00:59:32,180
love to create just as
much as they love to play.

1219
00:59:32,180 --> 00:59:34,890
And giving them the flexibility
that they can go in do,

1220
00:59:34,890 --> 00:59:37,560
it's an open sandbox,
do whatever you want.

1221
00:59:37,560 --> 00:59:40,360
We've maybe put a few
walls and barriers around.

1222
00:59:40,360 --> 00:59:43,170
But I would love to be
able to put into Madden,

1223
00:59:43,170 --> 00:59:46,590
hey, let's tweak the gravity.

1224
00:59:46,590 --> 00:59:49,960
Let's really change
up the physics.

1225
00:59:49,960 --> 00:59:53,840
Now, the core of madness is
about authenticity in realism.

1226
00:59:53,840 --> 00:59:55,900
So there's certain things
that we're not going

1227
00:59:55,900 --> 00:59:57,530
to deviate from that core.

1228
00:59:57,530 --> 00:59:59,730
But what little
flexibility we can

1229
00:59:59,730 --> 01:00:03,080
put in where you can break
Madden and have a little fun.

1230
01:00:03,080 --> 01:00:05,090
Because ultimately it's games.

1231
01:00:05,090 --> 01:00:08,680
You want to have fun and
that's what you want to do.

1232
01:00:08,680 --> 01:00:11,560
Personally I love
to see that we've

1233
01:00:11,560 --> 01:00:14,542
got that support in there.

1234
01:00:14,542 --> 01:00:19,490
And that's where the tiny
titan that slipped through.

1235
01:00:19,490 --> 01:00:21,300
That wasn't intentional.

1236
01:00:21,300 --> 01:00:24,560
I think we later just sort
of having fun and laughing

1237
01:00:24,560 --> 01:00:28,310
at ourselves we went ahead
released a ultimate team card

1238
01:00:28,310 --> 01:00:33,560
pack that allowed you to unlock
that as an intentional bug.

1239
01:00:33,560 --> 01:00:37,480
But the original case, it was
actually another one of those

1240
01:00:37,480 --> 01:00:43,380
we make a change and in our
excitement about getting that

1241
01:00:43,380 --> 01:00:46,395
change out to our
gamers, we move fast,

1242
01:00:46,395 --> 01:00:48,520
maybe don't have all the
processes in place that we

1243
01:00:48,520 --> 01:00:49,590
should've.

1244
01:00:49,590 --> 01:00:53,650
Someone does some rough testing,
looks good, push it out,

1245
01:00:53,650 --> 01:00:55,740
and something slips through.

1246
01:00:55,740 --> 01:00:57,330
But it's a big game.

1247
01:00:59,950 --> 01:01:02,487
When you start talking
about that program testing

1248
01:01:02,487 --> 01:01:04,070
and verification and
validation, there

1249
01:01:04,070 --> 01:01:06,281
are so many different
permutations

1250
01:01:06,281 --> 01:01:07,280
that have to be covered.

1251
01:01:07,280 --> 01:01:10,010
So many different modes.

1252
01:01:10,010 --> 01:01:14,040
It's not infinite, but it's
a very large finite number.

1253
01:01:14,040 --> 01:01:16,480
And it's almost impossible
to test everything,

1254
01:01:16,480 --> 01:01:18,990
which is why we're
trying to start

1255
01:01:18,990 --> 01:01:21,900
to get to a lot more
rigor in our testing

1256
01:01:21,900 --> 01:01:25,700
and how we measure and verify
that everything is on track.

1257
01:01:29,640 --> 01:01:31,300
Let's go ahead and take a break.

1258
01:01:31,300 --> 01:01:34,380
And I think, like I said, if you
have a question that you just

1259
01:01:34,380 --> 01:01:36,760
want to ask individually,
I'll be here

1260
01:01:36,760 --> 01:01:38,770
to talk a little bit longer.

1261
01:01:38,770 --> 01:01:40,669
So thank you.

1262
01:01:40,669 --> 01:01:43,483
[APPLAUSE]

1263
01:01:47,240 --> 01:01:49,310
PROFESSOR: So the
clock says 2:08.

1264
01:01:49,310 --> 01:01:50,722
Back in this room at 2:18.

1265
01:01:50,722 --> 01:01:52,680
We're going to give you
time to start planning.

1266
01:01:52,680 --> 01:01:54,500
You're going to run two tests.

1267
01:01:54,500 --> 01:01:56,930
And then you're going to
have a little bit of time

1268
01:01:56,930 --> 01:01:59,569
to reflect on your test
and just a really quick

1269
01:01:59,569 --> 01:02:00,860
tell us what you learned today.

1270
01:02:03,430 --> 01:02:08,690
OK, it's 2:18 or thereabouts.

1271
01:02:08,690 --> 01:02:12,080
Schedule for today is
you've got about 15 minutes

1272
01:02:12,080 --> 01:02:14,330
to do planning.

1273
01:02:14,330 --> 01:02:17,320
That means you should
know what observations

1274
01:02:17,320 --> 01:02:20,520
you're going to make when
you're doing these tests.

1275
01:02:20,520 --> 01:02:24,420
If you're going to do questions,
either creating a quick survey

1276
01:02:24,420 --> 01:02:27,900
or having questions you're
going to ask your testers.

1277
01:02:27,900 --> 01:02:30,545
Just like we asked
for last time.

1278
01:02:30,545 --> 01:02:32,670
You're going to spend 30
minutes doing a play test.

1279
01:02:32,670 --> 01:02:34,044
That means you're
going to set up

1280
01:02:34,044 --> 01:02:39,990
at least three computers
with three observers

1281
01:02:39,990 --> 01:02:42,890
to run those tests.

1282
01:02:42,890 --> 01:02:45,100
At 3:05 I'm going
to say, hey, switch.

1283
01:02:45,100 --> 01:02:48,280
That means anybody who's
doing an observation

1284
01:02:48,280 --> 01:02:51,030
is switching with the people
who are doing testing.

1285
01:02:51,030 --> 01:02:54,740
Everyone should have a chance
to both observe and play

1286
01:02:54,740 --> 01:02:58,074
at least three or four games.

1287
01:02:58,074 --> 01:02:59,490
That lasts for
another 30 minutes.

1288
01:02:59,490 --> 01:03:02,470
At 3:35, you're going
to time in teams

1289
01:03:02,470 --> 01:03:04,199
to complete your
focus test reports.

1290
01:03:04,199 --> 01:03:05,990
And then we'll just
ask you a few questions

1291
01:03:05,990 --> 01:03:09,560
about what you observed, but
no formal presentation today.

1292
01:03:09,560 --> 01:03:12,411
We just want to kind of
know what you saw today.

1293
01:03:12,411 --> 01:03:12,910
Questions?

1294
01:03:15,730 --> 01:03:16,700
All right.

1295
01:03:16,700 --> 01:03:20,240
So I'm going to yell at
2:35 to have your play

1296
01:03:20,240 --> 01:03:23,510
tests ready to start by 2:35.