1 00:00:00,050 --> 00:00:02,470 The following content is provided under a Creative 2 00:00:02,470 --> 00:00:03,880 Commons license. 3 00:00:03,880 --> 00:00:06,920 Your support will help MIT OpenCourseWare continue to 4 00:00:06,920 --> 00:00:10,570 offer high quality educational resources for free. 5 00:00:10,570 --> 00:00:13,470 To make a donation or view additional materials from 6 00:00:13,470 --> 00:00:19,290 hundreds of MIT courses, visit MIT OpenCourseWare at 7 00:00:19,290 --> 00:00:20,720 ocw.mit.edu. 8 00:00:20,720 --> 00:00:23,919 PROFESSOR: One example will be what happens when the battery 9 00:00:23,919 --> 00:00:26,100 runs up on the computer and what bubbles 10 00:00:26,100 --> 00:00:27,500 up when it's running. 11 00:00:27,500 --> 00:00:29,520 And there's certain computers that might [UNINTELLIGIBLE] 12 00:00:29,520 --> 00:00:30,050 hard drives. 13 00:00:30,050 --> 00:00:34,090 Some computers just keep it in RAM when you restart it. 14 00:00:34,090 --> 00:00:35,670 There's no big difference. 15 00:00:35,670 --> 00:00:40,188 What happens when an IM window pops up in the middle of the 16 00:00:40,188 --> 00:00:43,667 game and leaves the control screen? 17 00:00:43,667 --> 00:00:46,152 And all the keys start [UNINTELLIGIBLE] 18 00:00:46,152 --> 00:00:48,140 IM window. 19 00:00:48,140 --> 00:00:50,836 Ideally, they give you pop ups that doesn't work. 20 00:00:50,836 --> 00:00:54,730 But catching those is one of our favorite things that we do 21 00:00:54,730 --> 00:00:59,032 when we're making digital games here in the lab is like 22 00:00:59,032 --> 00:01:01,260 slamming all of our fingers on the keyboard at once and see 23 00:01:01,260 --> 00:01:02,860 what happens. 24 00:01:02,860 --> 00:01:07,250 And we're kind of looking for those things. 25 00:01:07,250 --> 00:01:11,650 We're also looking for subtler things like is the amount of 26 00:01:11,650 --> 00:01:13,342 RAM that those games are taking up in the computer 27 00:01:13,342 --> 00:01:17,240 slowly going up all the time? 28 00:01:17,240 --> 00:01:21,090 And of course, comparing it against a spec grade. 29 00:01:21,090 --> 00:01:25,470 So a character is supposed to be able to create a certain 30 00:01:25,470 --> 00:01:28,120 file hold style or something when you enter this key 31 00:01:28,120 --> 00:01:33,865 sequence, and 99 times out of 100 that works and that 1 out 32 00:01:33,865 --> 00:01:37,110 of 100 doesn't and we need to figure out why. 33 00:01:37,110 --> 00:01:37,670 That's our feat. 34 00:01:37,670 --> 00:01:40,920 So in a lot of those situations, what happens is 35 00:01:40,920 --> 00:01:46,850 that output comes back in the form of other parts and-- how 36 00:01:46,850 --> 00:01:50,092 many of you actually do programming? 37 00:01:50,092 --> 00:01:52,570 Oh, OK. 38 00:01:52,570 --> 00:01:55,935 How many of you have found a book report? 39 00:01:55,935 --> 00:01:57,753 OK, not that many. 40 00:02:00,770 --> 00:02:05,130 There are a couple of best practices where you follow a 41 00:02:05,130 --> 00:02:08,583 book report and this one translates a little bit into 42 00:02:08,583 --> 00:02:11,290 when you're really working in a team in a game. 43 00:02:11,290 --> 00:02:15,430 How do you communicate that something is wrong to someone 44 00:02:15,430 --> 00:02:18,850 else who is also responsible for the game, but may not 45 00:02:18,850 --> 00:02:22,500 necessarily have seen the problem? 46 00:02:22,500 --> 00:02:26,630 First of all, you describe what you were expecting to 47 00:02:26,630 --> 00:02:28,550 happen and what you were doing. 48 00:02:28,550 --> 00:02:32,795 I was expecting to go right when I push the right button. 49 00:02:32,795 --> 00:02:34,070 And then you write down what happened. 50 00:02:34,070 --> 00:02:38,410 You try to get as much detail as possible for the person to 51 00:02:38,410 --> 00:02:40,230 re-release this problem. 52 00:02:40,230 --> 00:02:45,660 Now, when it comes to war games, in many situations, the 53 00:02:45,660 --> 00:02:47,480 kind of detail that you need to provide largely comes up 54 00:02:47,480 --> 00:02:48,720 against it. 55 00:02:48,720 --> 00:02:53,160 So 'a' had this much money, 'b' had that much money, they 56 00:02:53,160 --> 00:02:57,010 try to do this game mechanic and no one knew what to do 57 00:02:57,010 --> 00:03:00,085 because the way the rules are written there's only one way 58 00:03:00,085 --> 00:03:02,450 to interpret it. 59 00:03:02,450 --> 00:03:06,590 And that gives your team enough information to be able 60 00:03:06,590 --> 00:03:08,920 to start acting on it. 61 00:03:08,920 --> 00:03:12,150 Because you are bargaining and your rules are kind of right 62 00:03:12,150 --> 00:03:15,330 out there in the open, there's usually less of a mystery to 63 00:03:15,330 --> 00:03:16,810 see where the problem occurs. 64 00:03:16,810 --> 00:03:19,470 But you know there's problem and it could end and you 65 00:03:19,470 --> 00:03:20,060 largely see where it lies. 66 00:03:20,060 --> 00:03:22,590 You may not know what to do to correct it, and that's 67 00:03:22,590 --> 00:03:25,320 actually where beta testing comes in. 68 00:03:25,320 --> 00:03:29,160 So but I'll get to the testing shortly. 69 00:03:29,160 --> 00:03:32,935 But technical testing, a lot of it again is comparing a 70 00:03:32,935 --> 00:03:33,950 given spec. 71 00:03:33,950 --> 00:03:35,382 Sometimes that spec is created from your team. 72 00:03:35,382 --> 00:03:36,330 This is what we want our game to do. 73 00:03:36,330 --> 00:03:40,340 This is what we expected to happen when we made this game. 74 00:03:40,340 --> 00:03:44,280 We expect that mid in the game, a certain game mechanic 75 00:03:44,280 --> 00:03:48,710 will start to take priority over other mechanics. 76 00:03:48,710 --> 00:03:51,900 Early on, everyone's buying property is a monopoly. 77 00:03:51,900 --> 00:03:56,990 Later on, everyone's going to be building houses and then 78 00:03:56,990 --> 00:03:58,910 even at the end, people start to mortgage stuff. 79 00:03:58,910 --> 00:04:00,920 That's what we kind of expected out of it. 80 00:04:00,920 --> 00:04:03,630 And then you play the game and you start realizing you 81 00:04:03,630 --> 00:04:06,400 mortgaging something way too early here, something's out of 82 00:04:06,400 --> 00:04:07,700 whack here. 83 00:04:07,700 --> 00:04:09,910 So that's the kind of technical 84 00:04:09,910 --> 00:04:10,780 testing you do in games. 85 00:04:10,780 --> 00:04:15,340 This is what we expect, why isn't it happening? 86 00:04:15,340 --> 00:04:19,940 The other side over here is kind of like being the 87 00:04:19,940 --> 00:04:23,750 defender of the player, being the player's advocate. 88 00:04:26,360 --> 00:04:30,130 How, can you be the person on the team that is actively 89 00:04:30,130 --> 00:04:34,516 looking for things in your game that are going to hurt 90 00:04:34,516 --> 00:04:36,648 your players and then mentioning it and saying, 91 00:04:36,648 --> 00:04:38,395 maybe we should do something about that. 92 00:04:38,395 --> 00:04:42,810 It's as simple as, here's the information that we want our 93 00:04:42,810 --> 00:04:44,760 players to have at a certain time. 94 00:04:44,760 --> 00:04:47,050 Here's the game state. 95 00:04:47,050 --> 00:04:49,790 Your characters have a certain state. 96 00:04:49,790 --> 00:04:52,790 They are this close to winning, but are this far away 97 00:04:52,790 --> 00:04:55,120 from winning to have these results here. 98 00:04:55,120 --> 00:04:57,445 Do the players understand that? 99 00:04:57,445 --> 00:05:00,730 Have you provided them with enough information to be able 100 00:05:00,730 --> 00:05:04,340 to make a good decision? 101 00:05:04,340 --> 00:05:06,145 If they know the game state, do they know what options they 102 00:05:06,145 --> 00:05:09,380 have in front of them? 103 00:05:09,380 --> 00:05:13,730 So as we play more and more board games this semester, you 104 00:05:13,730 --> 00:05:19,620 can actually look at the way how rules are presented 105 00:05:19,620 --> 00:05:19,870 redundantly. 106 00:05:19,870 --> 00:05:23,510 Sometimes they're written up in the rule sheet, but and of 107 00:05:23,510 --> 00:05:27,400 course you've got your index card-sized thing which goes 108 00:05:27,400 --> 00:05:29,270 through step one, two, three, four, these are the four 109 00:05:29,270 --> 00:05:31,238 things you've got to remember every single turn. 110 00:05:31,238 --> 00:05:33,581 You've got to get it, and everybody has a copy right in 111 00:05:33,581 --> 00:05:37,000 front of you because somebody on the team decided that that 112 00:05:37,000 --> 00:05:38,980 was a good idea. 113 00:05:38,980 --> 00:05:42,495 How did they come to that conclusion? 114 00:05:42,495 --> 00:05:47,170 Occasionally, they are doing it because they see there's a 115 00:05:47,170 --> 00:05:48,390 problem right away. 116 00:05:48,390 --> 00:05:52,060 There's too much information, and you just don't expect any 117 00:05:52,060 --> 00:05:54,690 player is ever going to be able to grasp all this 118 00:05:54,690 --> 00:05:55,485 information. 119 00:05:55,485 --> 00:06:00,423 So that's come out with a couple of ways to simplify it. 120 00:06:00,423 --> 00:06:03,734 Sometimes that involves taking up the mechanics, making 121 00:06:03,734 --> 00:06:07,750 something that was written up a text graphic report and some 122 00:06:07,750 --> 00:06:09,846 icons, reducing all the numbers by two so the math 123 00:06:09,846 --> 00:06:13,060 becomes easier. 124 00:06:13,060 --> 00:06:18,200 But sometimes, you do it by usability testing, and that's 125 00:06:18,200 --> 00:06:21,260 one kind of testing that often comes 126 00:06:21,260 --> 00:06:23,640 together with beta testing. 127 00:06:23,640 --> 00:06:26,850 I've invited Sara Verrilli, our development director, to 128 00:06:26,850 --> 00:06:30,709 come in here and talk about the kind of pre-testing that 129 00:06:30,709 --> 00:06:35,699 you can do, which is illustrated by this picture. 130 00:06:38,700 --> 00:06:42,500 And the book goes through several different kinds of 131 00:06:42,500 --> 00:06:45,410 environments where you do pre-testing that's sitting 132 00:06:45,410 --> 00:06:48,625 one-on-one with people and doing site 133 00:06:48,625 --> 00:06:51,320 testing for the group. 134 00:06:51,320 --> 00:06:54,422 Because this style is all about board games and card 135 00:06:54,422 --> 00:06:57,150 games, for the most part, we're designing games for 136 00:06:57,150 --> 00:06:58,050 three or four people to play at a time. 137 00:06:58,050 --> 00:07:01,440 The chances are, almost every single test that you're going 138 00:07:01,440 --> 00:07:03,016 to do [UNINTELLIGIBLE] at a single time. 139 00:07:03,016 --> 00:07:04,674 That's just the environment you're in 140 00:07:04,674 --> 00:07:05,570 that's meant to be played. 141 00:07:05,570 --> 00:07:09,390 That's why you end up testing it. 142 00:07:09,390 --> 00:07:11,056 Obviously, a single-play computer game, that's not 143 00:07:11,056 --> 00:07:13,800 necessarily the environment you're testing in. 144 00:07:13,800 --> 00:07:19,985 Actually, before I transition over, there is one thing I 145 00:07:19,985 --> 00:07:21,670 want to mention about technical testing. 146 00:07:21,670 --> 00:07:23,855 Sometimes the specs you're testing against are not the 147 00:07:23,855 --> 00:07:26,710 ones that came up that would necessarily come up 148 00:07:26,710 --> 00:07:28,640 with phi over t. 149 00:07:28,640 --> 00:07:30,520 If you're doing a computer game-- 150 00:07:30,520 --> 00:07:34,510 say for an Xbox or a PS3 or a Wii. 151 00:07:34,510 --> 00:07:37,001 There are these things called technical [? evaluation ?] 152 00:07:37,001 --> 00:07:42,280 requirements, which are basically huge lists of things 153 00:07:42,280 --> 00:07:46,391 that your game must do before Microsoft will sign your title 154 00:07:46,391 --> 00:07:50,720 or allow it to be released under that title. 155 00:07:50,720 --> 00:07:54,796 PC obviously doesn't have this, but if you release it on 156 00:07:54,796 --> 00:07:57,900 speed or something, then chances are, you'll have 157 00:07:57,900 --> 00:07:58,110 requirements. 158 00:07:58,110 --> 00:07:59,730 It could be something as simple as generally, it 159 00:07:59,730 --> 00:08:02,596 doesn't crash-- 160 00:08:02,596 --> 00:08:06,830 and games do crash, even on PS3 and Xbox-- 161 00:08:06,830 --> 00:08:12,330 you get things like what happens when you're playing a 162 00:08:12,330 --> 00:08:14,442 game on the PS3 or Xbox-- 163 00:08:14,442 --> 00:08:18,012 I guess more the PS2 and the first Xbox-- and someone yanks 164 00:08:18,012 --> 00:08:20,880 out the cable? 165 00:08:20,880 --> 00:08:23,510 All you're thinking is that you're-- and then the keyboard 166 00:08:23,510 --> 00:08:25,805 comes out, what happens? 167 00:08:25,805 --> 00:08:28,390 Does it continue? 168 00:08:28,390 --> 00:08:29,740 STUDENT: It buzzes. 169 00:08:29,740 --> 00:08:31,900 PROFESSOR: It goes into pause, right? 170 00:08:31,900 --> 00:08:35,289 And usually it tells you, your controller is disconnected. 171 00:08:35,289 --> 00:08:36,950 And that's a requirement. 172 00:08:36,950 --> 00:08:40,111 Someone had to write that code specifically to detect that 173 00:08:40,111 --> 00:08:44,610 situation and then throughout the suite across the game. 174 00:08:44,610 --> 00:08:47,590 And the reason everybody had to do that was because 175 00:08:47,590 --> 00:08:49,560 Microsoft and Sony and Nintendo-- 176 00:08:49,560 --> 00:08:50,730 well, not so much Nintendo nowadays 177 00:08:50,730 --> 00:08:53,230 because Nintendo is wireless-- 178 00:08:53,230 --> 00:08:57,880 but these are the kinds of requirements that they expect. 179 00:08:57,880 --> 00:09:01,248 If you lose your connection to Xbox live halfway through the 180 00:09:01,248 --> 00:09:03,650 game then something's going to happen. 181 00:09:03,650 --> 00:09:08,595 And so again, for that kind of technical testing, you're sort 182 00:09:08,595 --> 00:09:11,416 of comparing it against spec, but spec may not necessarily 183 00:09:11,416 --> 00:09:14,392 be something that you're keeping up with. 184 00:09:14,392 --> 00:09:18,860 It may be something that the publisher or your platform 185 00:09:18,860 --> 00:09:21,230 developer, platform manufacturer mandated. 186 00:09:21,230 --> 00:09:23,750 How does it affect the board games is that sometimes, board 187 00:09:23,750 --> 00:09:27,225 games are actually going to require half the 188 00:09:27,225 --> 00:09:28,480 requirements to that. 189 00:09:28,480 --> 00:09:31,543 If you're responsible for the design of the box, for 190 00:09:31,543 --> 00:09:33,930 instance, there are things they're going to require on 191 00:09:33,930 --> 00:09:38,046 the outside of the box, like what ages are the game 192 00:09:38,046 --> 00:09:40,820 applicable for, how long is it meant to be played, how many 193 00:09:40,820 --> 00:09:43,680 people are going to be playing your game. 194 00:09:43,680 --> 00:09:49,800 I think Days of Wonder, which is a publisher, actually 195 00:09:49,800 --> 00:09:54,370 requires a character to be on the box. 196 00:09:54,370 --> 00:09:58,120 Even if your game has no characters, there must be a 197 00:09:58,120 --> 00:10:01,750 human-like presence on the box or your players are on the box 198 00:10:01,750 --> 00:10:05,010 because they think that it increases sales, and they 199 00:10:05,010 --> 00:10:06,720 probably have the numbers to back it up. 200 00:10:06,720 --> 00:10:10,180 That's why if you look at things like Ticket for Ride, 201 00:10:10,180 --> 00:10:12,168 for instance, has people. 202 00:10:12,168 --> 00:10:16,450 If you play Ticket to Ride, it has no people in it, but it 203 00:10:16,450 --> 00:10:19,940 has people on the box because they want people to think 204 00:10:19,940 --> 00:10:21,335 that's what users are. 205 00:10:25,070 --> 00:10:31,397 So that's just an example of things that a board game 206 00:10:31,397 --> 00:10:32,790 publisher may require you to do. 207 00:10:32,790 --> 00:10:37,180 Very common that you'll have to do translations in a range 208 00:10:37,180 --> 00:10:40,920 of different translations for the different countries it's 209 00:10:40,920 --> 00:10:43,016 going to be released in. 210 00:10:43,016 --> 00:10:45,435 It may not be your job to do it, the publishers may end up 211 00:10:45,435 --> 00:10:47,027 being the ones helping you with the translation, getting 212 00:10:47,027 --> 00:10:49,260 all the publishing data formatted decently, but 213 00:10:49,260 --> 00:10:52,560 someone along the stage is making sure that happens 214 00:10:52,560 --> 00:10:55,274 before the game hits the shelf. 215 00:10:58,340 --> 00:11:00,618 OK, that's technical testing. 216 00:11:00,618 --> 00:11:05,956 That's a little bit about becoming a player and so let's 217 00:11:05,956 --> 00:11:09,372 talk about data center a bit. 218 00:11:09,372 --> 00:11:13,764 SARA VERRILLI: Let me bump you a little here, 219 00:11:13,764 --> 00:11:14,460 if you don't care. 220 00:11:14,460 --> 00:11:15,710 PROFESSOR: Oh, right. 221 00:11:37,466 --> 00:11:38,954 SARA VERRILLI: OK. 222 00:11:38,954 --> 00:11:39,946 Let's see. 223 00:11:39,946 --> 00:11:41,930 Let's actually go ahead. 224 00:11:45,898 --> 00:11:47,882 So focus testing. 225 00:11:47,882 --> 00:11:50,767 This is a very similar track for the summer program folks. 226 00:11:50,767 --> 00:11:53,840 I have about 10 minutes to go back and relook at my notes 227 00:11:53,840 --> 00:11:57,980 and relook at the lecture, so if it's a little stumbly in 228 00:11:57,980 --> 00:12:01,209 parts, it's because I haven't given it for a while and 229 00:12:01,209 --> 00:12:01,460 haven't thought about it. 230 00:12:01,460 --> 00:12:02,766 It's much more geared, the original talk is much more 231 00:12:02,766 --> 00:12:05,140 geared towards computer testing, and testing 232 00:12:05,140 --> 00:12:08,890 specifically single-player computer games than it's 233 00:12:08,890 --> 00:12:12,770 testing board games, which are almost a multi-user. 234 00:12:12,770 --> 00:12:16,598 So first of all, focus testing is actually dissimilar. 235 00:12:16,598 --> 00:12:20,370 Focus testing is only a specific case of user testing, 236 00:12:20,370 --> 00:12:22,720 which is to say, you're testing your game using actual 237 00:12:22,720 --> 00:12:25,910 users, not your development team, not people who play 238 00:12:25,910 --> 00:12:28,484 board games 12 hours a day because that's what they love 239 00:12:28,484 --> 00:12:31,100 to do all the time, but more the sort of average person who 240 00:12:31,100 --> 00:12:33,645 might want to pick up your game off the shelf, go buy it 241 00:12:33,645 --> 00:12:37,300 and then go play it with their family and then go out to 242 00:12:37,300 --> 00:12:40,198 dinner at Chuck E. Cheese or something. 243 00:12:40,198 --> 00:12:44,245 Another type of user tester is usability testing, which is 244 00:12:44,245 --> 00:12:46,770 fairly similar to focus testing in as much as you're 245 00:12:46,770 --> 00:12:48,360 still using users for it. 246 00:12:48,360 --> 00:12:50,860 Focus testing is usually much more geared towards are people 247 00:12:50,860 --> 00:12:52,073 enjoying your game? 248 00:12:52,073 --> 00:12:55,370 Do people want to play your game? 249 00:12:55,370 --> 00:12:59,060 Are people going through the sort of play experiences you 250 00:12:59,060 --> 00:13:01,290 want them to have as they play their game? 251 00:13:01,290 --> 00:13:02,920 Because usually when you create a game, you're 252 00:13:02,920 --> 00:13:05,410 thinking, oh, like for Settlers of Catan, you expect 253 00:13:05,410 --> 00:13:06,950 people to spend a lot of time talking with each other 254 00:13:06,950 --> 00:13:09,326 trading things back and forth. 255 00:13:09,326 --> 00:13:12,960 When you're making Diplomacy, you expect people to spend a 256 00:13:12,960 --> 00:13:16,200 lot of time arguing with each other over well, if you invade 257 00:13:16,200 --> 00:13:19,060 that country, I'll invade that country and we'll gang up on 258 00:13:19,060 --> 00:13:21,280 him and we'll all come out of the game and then we'll just 259 00:13:21,280 --> 00:13:25,098 divide four between the two of us and win while you're making 260 00:13:25,098 --> 00:13:25,618 secret underhanded arrangements with the other 261 00:13:25,618 --> 00:13:28,000 guys, but OK, so he's going to attack you on this ground, we 262 00:13:28,000 --> 00:13:32,330 both know that, that sort of thing, as opposed to, say, 263 00:13:32,330 --> 00:13:35,192 Transamerica which is just more putting pieces down and 264 00:13:35,192 --> 00:13:38,420 it's pretty much independent play in many ways. 265 00:13:41,420 --> 00:13:45,612 Usability testing is much more about how you easy is it for 266 00:13:45,612 --> 00:13:46,920 people to play your game. 267 00:13:46,920 --> 00:13:49,150 Do they understand your rules? 268 00:13:49,150 --> 00:13:50,490 Are the mechanics clear? 269 00:13:50,490 --> 00:13:53,100 When they sit down and start sorting out the tokens, can 270 00:13:53,100 --> 00:13:57,055 they tell which tokens are the player pieces and which cards 271 00:13:57,055 --> 00:13:58,056 are which and how you use those? 272 00:13:58,056 --> 00:14:01,220 Do the icons on your game make sense to people? 273 00:14:01,220 --> 00:14:02,570 Do they understand them? 274 00:14:02,570 --> 00:14:03,020 Things like that. 275 00:14:03,020 --> 00:14:06,162 It's not really about whether or not people like your game 276 00:14:06,162 --> 00:14:07,400 or are enjoying your game. 277 00:14:07,400 --> 00:14:11,056 It's about can people understand your game and start 278 00:14:11,056 --> 00:14:12,306 playing it. 279 00:14:14,500 --> 00:14:19,545 So here we go. 280 00:14:19,545 --> 00:14:23,703 One big problem we run into when people sit down and we 281 00:14:23,703 --> 00:14:25,385 have people playing your game and giving lots of comments, 282 00:14:25,385 --> 00:14:27,670 this was really fun or this really sucked, or I hate this 283 00:14:27,670 --> 00:14:29,080 part where I have to spend time talking to the other 284 00:14:29,080 --> 00:14:33,120 players, I want to be able to stare at just my cards and 285 00:14:33,120 --> 00:14:36,350 board completely without interacting with anyone else. 286 00:14:36,350 --> 00:14:37,710 Focus testers will give you lots and lots of feedback, 287 00:14:37,710 --> 00:14:41,140 some of it is useful, some of it is not. 288 00:14:41,140 --> 00:14:45,986 And you need to remember that focus testers are not actually 289 00:14:45,986 --> 00:14:46,890 people building the game. 290 00:14:46,890 --> 00:14:47,740 Focus testers are the people who you are trying to get 291 00:14:47,740 --> 00:14:52,020 information about your game from and one problem that we 292 00:14:52,020 --> 00:14:56,890 often see in experienced game developers and people making 293 00:14:56,890 --> 00:15:01,560 games doing is, oh, the focus tester said we have to make 294 00:15:01,560 --> 00:15:06,840 the whole game red, quick, let's make everything red. 295 00:15:06,840 --> 00:15:11,180 So and that's usually, that's not a very 296 00:15:11,180 --> 00:15:11,957 good solution, right? 297 00:15:11,957 --> 00:15:13,780 Who knows what the problem with this actually was, but 298 00:15:13,780 --> 00:15:16,970 what you need to do is find out why did the players want 299 00:15:16,970 --> 00:15:18,010 to make everything red? 300 00:15:18,010 --> 00:15:19,550 Is it because they don't like the artistic 301 00:15:19,550 --> 00:15:21,858 design of the game? 302 00:15:21,858 --> 00:15:22,340 Is it because someone in the game is colorblind? 303 00:15:22,340 --> 00:15:26,196 Someone playing the game currently is colorblind, is 304 00:15:26,196 --> 00:15:29,088 having a hard time telling things apart so they'd like 305 00:15:29,088 --> 00:15:31,304 everything to be the same color so everyone's having the 306 00:15:31,304 --> 00:15:32,275 same problem they are. 307 00:15:32,275 --> 00:15:33,250 I mean, who knows? 308 00:15:33,250 --> 00:15:37,720 It's hard to tell what people are thinking. 309 00:15:37,720 --> 00:15:40,320 When they give you advice, and when they give you these 310 00:15:40,320 --> 00:15:41,560 comments, you have to have some way to filter those 311 00:15:41,560 --> 00:15:44,260 comments, some way to think about them and some way to 312 00:15:44,260 --> 00:15:48,500 decide what actually is the right reaction to take in view 313 00:15:48,500 --> 00:15:52,525 of what you want your game to do and how you want people to 314 00:15:52,525 --> 00:15:52,660 play the game. 315 00:15:52,660 --> 00:15:54,887 Since you're the game designers and developers, you 316 00:15:54,887 --> 00:15:57,100 have to make those decisions. 317 00:15:57,100 --> 00:16:00,014 Focus testers can only give you data. 318 00:16:00,014 --> 00:16:02,882 You shouldn't ask them to design your game for you. 319 00:16:02,882 --> 00:16:05,750 Maybe you actually have good ideas that you want to use, 320 00:16:05,750 --> 00:16:08,316 but you want to make sure that they're good ideas that you do 321 00:16:08,316 --> 00:16:10,800 want to use. 322 00:16:10,800 --> 00:16:16,866 So you may notice, these are both my lectures, and these 323 00:16:16,866 --> 00:16:20,170 are pictures that Mike [? Dropho ?] drew for a 324 00:16:20,170 --> 00:16:21,720 poster, and we love him so much. 325 00:16:21,720 --> 00:16:25,150 So in the ideal focus testing situation, you're really 326 00:16:25,150 --> 00:16:29,090 trying to get as much data as you can about your player 327 00:16:29,090 --> 00:16:32,765 playing the game without spoiling the information. 328 00:16:32,765 --> 00:16:36,060 What you'd really like to recreate in a lot of ways is, 329 00:16:36,060 --> 00:16:37,826 so you just bought the game and have 330 00:16:37,826 --> 00:16:38,610 gotten home, what happens? 331 00:16:38,610 --> 00:16:43,100 And that's what this picture is all about, there's somebody 332 00:16:43,100 --> 00:16:47,240 luring somebody to play the game without any information. 333 00:16:53,117 --> 00:17:03,995 So things to think about when you're planning your focus 334 00:17:03,995 --> 00:17:09,520 testing is, what are you testing? 335 00:17:09,520 --> 00:17:11,900 What questions do you want to ask? 336 00:17:11,900 --> 00:17:13,328 What information do you want to get out of it? 337 00:17:13,328 --> 00:17:15,940 How are you going to test it? 338 00:17:15,940 --> 00:17:18,490 How many players do you need? 339 00:17:18,490 --> 00:17:21,170 What kind of interactions do you want to see? 340 00:17:21,170 --> 00:17:24,554 On the backhand, who are you testing with? 341 00:17:24,554 --> 00:17:26,582 What kind of people are going to make your game? 342 00:17:26,582 --> 00:17:30,912 Are you making a game for children age 9 to 15? 343 00:17:30,912 --> 00:17:33,407 Are you making it for college-age students? 344 00:17:33,407 --> 00:17:37,648 Are you making a game that you think that older players will 345 00:17:37,648 --> 00:17:38,896 want to play? 346 00:17:38,896 --> 00:17:41,391 Because if you're making a game that you think is going 347 00:17:41,391 --> 00:17:43,387 to be popular with the senior citizens market, testing it 348 00:17:43,387 --> 00:17:45,882 with a bunch of elementary school kids probably isn't 349 00:17:45,882 --> 00:17:48,876 going to go very will. 350 00:17:48,876 --> 00:17:52,312 Trivial Pursuit Classic Edition, a game that came out 351 00:17:52,312 --> 00:17:56,130 about 20-odd years ago, I suspect that if you sat down a 352 00:17:56,130 --> 00:18:00,082 group of high school students with it and had them play it, 353 00:18:00,082 --> 00:18:00,786 about three quarters of the questions they'd just stare at 354 00:18:00,786 --> 00:18:05,022 you and be like, I've never heard of that. 355 00:18:05,022 --> 00:18:09,468 So, that's certainly where the audience comes in. 356 00:18:09,468 --> 00:18:14,408 So, the what, planning out the questions that you think 357 00:18:14,408 --> 00:18:15,396 you're going to get information for. 358 00:18:15,396 --> 00:18:19,350 When you know what information for focus testing, it's a lot 359 00:18:19,350 --> 00:18:22,630 easier to know what to keep track of, how to take notes, 360 00:18:22,630 --> 00:18:25,981 what reactions you're most interested in as people are 361 00:18:25,981 --> 00:18:27,238 playing the game. 362 00:18:27,238 --> 00:18:30,136 So, what questions are you going to ask? 363 00:18:30,136 --> 00:18:32,080 What data it takes to answer your question? 364 00:18:32,080 --> 00:18:33,930 And how are you going to get that data? 365 00:18:33,930 --> 00:18:35,220 Are you going to go all through 366 00:18:35,220 --> 00:18:36,780 observation taking notes? 367 00:18:36,780 --> 00:18:38,620 Are you going to just step in and out of questions? 368 00:18:38,620 --> 00:18:42,249 So why did you move that character from there, just out 369 00:18:42,249 --> 00:18:43,873 of curiosity I'm wondering why you made that move. 370 00:18:43,873 --> 00:18:46,308 Will you tell me about it? 371 00:18:46,308 --> 00:18:49,230 Or, you can ask them to fill out a questionnaire 372 00:18:49,230 --> 00:18:49,717 at the end of it. 373 00:18:49,717 --> 00:18:53,126 Depending on what data you want, different 374 00:18:53,126 --> 00:18:54,610 methods will work best. 375 00:18:54,610 --> 00:19:01,684 In general, usually the best way to do it is observation. 376 00:19:01,684 --> 00:19:05,636 As soon as you ask someone questions, you sort of perturb 377 00:19:05,636 --> 00:19:08,366 the system and when you ask them to fill out a survey, 378 00:19:08,366 --> 00:19:09,930 it's after they've done it. 379 00:19:09,930 --> 00:19:12,030 They've kind of forgotten what they were doing. 380 00:19:12,030 --> 00:19:14,520 They may not exactly remember, and to be perfectly honest, 381 00:19:14,520 --> 00:19:15,530 people would really rather play your game 382 00:19:15,530 --> 00:19:17,140 than fill out a form. 383 00:19:17,140 --> 00:19:21,650 So they'll be more engaged more interested and more 384 00:19:21,650 --> 00:19:24,606 interactive when they're playing a game and that's when 385 00:19:24,606 --> 00:19:25,770 you're going to get your best data. 386 00:19:25,770 --> 00:19:28,190 But there are times when you want to ask someone something 387 00:19:28,190 --> 00:19:29,920 sort of analytical and you don't want someone to be 388 00:19:29,920 --> 00:19:32,910 thinking analytically while they're playing the game in a 389 00:19:32,910 --> 00:19:35,614 lot of ways , and so there are times when that reflection 390 00:19:35,614 --> 00:19:39,125 comes out so much better and it's better to have that 391 00:19:39,125 --> 00:19:40,320 written down and get it in your own words. 392 00:19:40,320 --> 00:19:44,505 STUDENT: You would hope it plays better than a survey. 393 00:19:44,505 --> 00:19:46,920 SARA VERRILLI: If your game is better than a survey, you 394 00:19:46,920 --> 00:19:51,358 probably already observed a problem right there and you're 395 00:19:51,358 --> 00:19:53,310 probably glad you found it. 396 00:19:57,230 --> 00:19:58,390 So how? 397 00:19:58,390 --> 00:20:01,030 In order to get all that data, you're going to have to 398 00:20:01,030 --> 00:20:03,250 actually interact with the user, someone who's never seen 399 00:20:03,250 --> 00:20:06,300 your game before, someone who doesn't actually care whether 400 00:20:06,300 --> 00:20:08,550 or not you get good information out of 401 00:20:08,550 --> 00:20:09,250 playing this game. 402 00:20:09,250 --> 00:20:11,330 They came here to sit down and play the game, and while they 403 00:20:11,330 --> 00:20:14,260 would like to help you by giving you information, that's 404 00:20:14,260 --> 00:20:17,030 not their primary goal in a lot of ways. 405 00:20:17,030 --> 00:20:19,074 Usually, their primary goal is to sit down, and play the 406 00:20:19,074 --> 00:20:19,190 game, and have a good time. 407 00:20:19,190 --> 00:20:21,610 So you need to be able to structure the interaction with 408 00:20:21,610 --> 00:20:25,210 them so that getting the data to you is as 409 00:20:25,210 --> 00:20:27,310 easy for them as possible. 410 00:20:27,310 --> 00:20:30,115 So we need to think about what this player needs before he or 411 00:20:30,115 --> 00:20:30,740 she sits down to play. 412 00:20:30,740 --> 00:20:32,464 Usually, in a board game or a card game, 413 00:20:32,464 --> 00:20:33,990 that's the set of rules. 414 00:20:33,990 --> 00:20:36,520 In a computer, it's always trickier. 415 00:20:36,520 --> 00:20:39,700 Usually when we're bringing in people to test the first 416 00:20:39,700 --> 00:20:42,690 prototypes, we have a game that works, but we don't have 417 00:20:42,690 --> 00:20:43,490 any instructions in game. 418 00:20:43,490 --> 00:20:45,390 If we're going to have a tutorial or anything like 419 00:20:45,390 --> 00:20:48,080 that, that's usually one of the last things that goes in. 420 00:20:48,080 --> 00:20:50,620 So when you're testing computer games early in the 421 00:20:50,620 --> 00:20:53,170 process, usually you end up sitting down and writing out, 422 00:20:53,170 --> 00:20:54,820 well, this is what the tutorial would look like if we 423 00:20:54,820 --> 00:20:57,120 had it, so read this and then sit down and play it. 424 00:20:57,120 --> 00:20:59,005 With board games and card games, it's actually a little 425 00:20:59,005 --> 00:21:01,610 bit easier, because you can say, a ha, this is the rules 426 00:21:01,610 --> 00:21:03,530 that are going to package with the games. 427 00:21:03,530 --> 00:21:05,810 Give you the rules, step back and see what happens. 428 00:21:05,810 --> 00:21:07,920 So, it's actually an easier aspect for 429 00:21:07,920 --> 00:21:10,330 testing board games. 430 00:21:10,330 --> 00:21:14,636 And finally, treat the players politely. 431 00:21:14,636 --> 00:21:16,995 When they find a misspelling in your rules or when they 432 00:21:16,995 --> 00:21:20,050 discover that you've left out six of the key cards to play 433 00:21:20,050 --> 00:21:23,005 the game, oh, yeah, you're right, thank you for pointing 434 00:21:23,005 --> 00:21:25,868 that out, I wouldn't have noticed that if you hadn't 435 00:21:25,868 --> 00:21:26,260 shown it to me. 436 00:21:26,260 --> 00:21:28,255 Whether you would have or not doesn't really matter. 437 00:21:28,255 --> 00:21:30,490 They're doing you a favor by playing your game that isn't 438 00:21:30,490 --> 00:21:33,460 quite working well, so you want them to know that their 439 00:21:33,460 --> 00:21:37,557 contribution is invaluable and you appreciate the time and 440 00:21:37,557 --> 00:21:38,807 effort they've put into it. 441 00:21:42,850 --> 00:21:46,110 A few things, the more they talk about your game and the 442 00:21:46,110 --> 00:21:48,390 mechanics they find that break or that do or don't work, 443 00:21:48,390 --> 00:21:51,482 that's all data right there, That's all great. 444 00:21:51,482 --> 00:21:54,077 Another great advantage of testing board games is you 445 00:21:54,077 --> 00:21:56,240 usually get groups of players, like two or three or four 446 00:21:56,240 --> 00:21:59,520 players, and when you have multiple players, they'll talk 447 00:21:59,520 --> 00:22:00,526 to each other about it. 448 00:22:00,526 --> 00:22:01,080 Oh, I noticed this. 449 00:22:01,080 --> 00:22:03,390 Did you notice this strategy? 450 00:22:03,390 --> 00:22:05,115 Yeah, I did, but I noticed that if you add that strategy 451 00:22:05,115 --> 00:22:07,780 that you can do this thing and this other thing and that's 452 00:22:07,780 --> 00:22:09,240 all really neat. 453 00:22:09,240 --> 00:22:10,926 When you're testing single-player computer games 454 00:22:10,926 --> 00:22:13,940 like we usually do in the summer program, you've got a 455 00:22:13,940 --> 00:22:15,990 single user that is sitting there between 456 00:22:15,990 --> 00:22:17,020 them and the computer. 457 00:22:17,020 --> 00:22:19,735 You have to watch and figure out what they're thinking. 458 00:22:19,735 --> 00:22:22,776 When you have a group, it's not usually that hard to get 459 00:22:22,776 --> 00:22:24,553 them talking about what they're doing and what they're 460 00:22:24,553 --> 00:22:27,347 thinking, and so that's a really good way to try 461 00:22:27,347 --> 00:22:29,073 and get more data. 462 00:22:34,020 --> 00:22:38,215 So yeah, here's sort of the end summary. 463 00:22:38,215 --> 00:22:40,100 Have a specific question and questions in mind. 464 00:22:40,100 --> 00:22:43,380 Know what you're doing in focus testing, know why you're 465 00:22:43,380 --> 00:22:44,330 doing usability testing. 466 00:22:44,330 --> 00:22:48,150 If you don't know what you're trying to test for, then your 467 00:22:48,150 --> 00:22:50,370 data is going to be a lot harder to figure 468 00:22:50,370 --> 00:22:51,220 out what to do with. 469 00:22:51,220 --> 00:22:53,810 So you always want to we have that question in mind. 470 00:22:53,810 --> 00:22:55,250 That's actually a really big one. 471 00:22:55,250 --> 00:22:59,974 What data you need to answer that question so you can 472 00:22:59,974 --> 00:23:03,040 collect it and know how to collect it, and make sure that 473 00:23:03,040 --> 00:23:05,750 you can give the information to the players, a standardized 474 00:23:05,750 --> 00:23:08,925 set of information to the players and step away so 475 00:23:08,925 --> 00:23:11,200 you're not sort of coloring the information and giving 476 00:23:11,200 --> 00:23:13,770 them the things you're talking about by, oh I forgot to 477 00:23:13,770 --> 00:23:17,290 mention, oh and did I say, so you've actually got sort of a 478 00:23:17,290 --> 00:23:23,635 standardized standard size play group scenario set down. 479 00:23:23,635 --> 00:23:26,605 OK, I think that's actually what [? I came for. ?] 480 00:23:29,580 --> 00:23:33,496 PROFESSOR: So a couple of words about when to be 481 00:23:33,496 --> 00:23:37,104 testing, which is really as soon as possible. 482 00:23:37,104 --> 00:23:40,540 Usually, before you think it's possible, you should be 483 00:23:40,540 --> 00:23:44,780 already starting to plan your fist test because at that 484 00:23:44,780 --> 00:23:47,228 point, you're probably going for the lowest-hanging fruit, 485 00:23:47,228 --> 00:23:50,216 which will be probably somebody else in this class 486 00:23:50,216 --> 00:23:53,900 because they're going to give you feedback anyways, you can 487 00:23:53,900 --> 00:23:56,340 use that favor in exchange. 488 00:23:56,340 --> 00:23:59,480 The whole point of making testing as simple as possible 489 00:23:59,480 --> 00:24:02,830 is you want to be able to get information while you still 490 00:24:02,830 --> 00:24:08,550 have time to fix it, to do something with that data. 491 00:24:08,550 --> 00:24:12,050 What has happened a lot of times in this class is we have 492 00:24:12,050 --> 00:24:15,650 a project team where they come up with a game requires say, 493 00:24:15,650 --> 00:24:20,100 generating so many cards or that they're going to spend 494 00:24:20,100 --> 00:24:23,150 most of the project deadline actually creating those cards. 495 00:24:23,150 --> 00:24:26,710 They'll test it, but they'll test it on themselves because 496 00:24:26,710 --> 00:24:28,830 the team already gets it and they're not 497 00:24:28,830 --> 00:24:29,720 looking for other people. 498 00:24:29,720 --> 00:24:34,696 They have a plan to ask their friends to come in for a 499 00:24:34,696 --> 00:24:35,374 specific time, they've set aside an hour or 500 00:24:35,374 --> 00:24:36,330 something like that. 501 00:24:36,330 --> 00:24:39,200 So what happens is that the last week comes around and 502 00:24:39,200 --> 00:24:41,270 they start playing it with people who haven't seen the 503 00:24:41,270 --> 00:24:44,020 game before and then discovered problems. 504 00:24:44,020 --> 00:24:45,970 Unfortunately, the game has 100 cards that all 505 00:24:45,970 --> 00:24:48,350 now need to be fixed. 506 00:24:48,350 --> 00:24:50,540 This is a problem because you just discovered 507 00:24:50,540 --> 00:24:52,826 that problem too late. 508 00:24:52,826 --> 00:24:54,730 A couple of ways to handle that, though, one, don't 509 00:24:54,730 --> 00:24:59,236 decide on a game where making changes across the entire game 510 00:24:59,236 --> 00:25:02,140 is going to take you more time than to test it because that's 511 00:25:02,140 --> 00:25:04,820 going to make it impossible to fix anything. 512 00:25:04,820 --> 00:25:06,910 But the other one is just testing early. 513 00:25:06,910 --> 00:25:10,335 As soon as you've got a couple of key mechanics down, sure 514 00:25:10,335 --> 00:25:12,580 you are planning to play this game with 100 515 00:25:12,580 --> 00:25:15,360 cards, start with 20. 516 00:25:15,360 --> 00:25:19,150 Start with a small of group of cards, hence your core can see 517 00:25:19,150 --> 00:25:20,540 people and understand that they're going to discover 518 00:25:20,540 --> 00:25:24,190 things in the production of those 20 cards and in the 519 00:25:24,190 --> 00:25:27,160 testing of those 20 cards that maybe you don't want to be 520 00:25:27,160 --> 00:25:31,790 doing for the next test and you can redo those 20 cards 521 00:25:31,790 --> 00:25:33,930 pretty quickly the way that you can't redo 100 cards. 522 00:25:33,930 --> 00:25:35,280 Just an example. 523 00:25:35,280 --> 00:25:38,530 I'm really not encouraging people to do a 100-card game, 524 00:25:38,530 --> 00:25:39,805 by the way. 525 00:25:39,805 --> 00:25:41,986 A regular poker deck, 52 cards, that's all you need for 526 00:25:41,986 --> 00:25:43,236 a really interesting game. 527 00:25:45,600 --> 00:25:47,760 Let's see. 528 00:25:47,760 --> 00:25:50,140 Some other things. 529 00:25:50,140 --> 00:25:53,520 Say you're doing a test, but it's not with 530 00:25:53,520 --> 00:25:54,880 your entire team present. 531 00:25:54,880 --> 00:25:59,620 Say it's just you and your teammates are all living in 532 00:25:59,620 --> 00:26:02,700 different dorms and you decided to play with 533 00:26:02,700 --> 00:26:03,120 those in your dorm. 534 00:26:03,120 --> 00:26:05,370 That's great, because it's a cool game and you've got your 535 00:26:05,370 --> 00:26:08,060 feedback, how do you communicate that 536 00:26:08,060 --> 00:26:09,020 back to your team? 537 00:26:09,020 --> 00:26:11,890 I talked a little bit about writing up a little report, 538 00:26:11,890 --> 00:26:14,976 but here's a couple of other tips because one thing you 539 00:26:14,976 --> 00:26:17,520 have to remember is that chances are, every single idea 540 00:26:17,520 --> 00:26:19,400 in the game, somebody in your team is probably really 541 00:26:19,400 --> 00:26:22,970 passionate about it and what you're basically trying to do 542 00:26:22,970 --> 00:26:25,733 is try to convince someone in your team that maybe this idea 543 00:26:25,733 --> 00:26:27,550 should be changed. 544 00:26:27,550 --> 00:26:30,370 And your game will probably be better for it, but there are 545 00:26:30,370 --> 00:26:33,470 some tactful was of telling that. 546 00:26:33,470 --> 00:26:36,900 So let's see. 547 00:26:36,900 --> 00:26:39,680 Keep it short, this also applies to book reports. 548 00:26:39,680 --> 00:26:42,580 I talked about you giving people enough information to 549 00:26:42,580 --> 00:26:46,710 reproduce what the problem was so that they can understand 550 00:26:46,710 --> 00:26:49,940 the state of the game and why the problem is occurring. 551 00:26:49,940 --> 00:26:53,380 And the flip side of that is not too much information, not 552 00:26:53,380 --> 00:26:56,320 more information than is necessary to keep it short. 553 00:26:56,320 --> 00:26:59,870 Little [? farm ?] codes, give them something that's easy for 554 00:26:59,870 --> 00:27:01,220 people to scan. 555 00:27:01,220 --> 00:27:04,149 Don't make them read and essay for them to figure out what 556 00:27:04,149 --> 00:27:06,250 comments will be solved. 557 00:27:06,250 --> 00:27:08,610 Now actually, one thing to keep in mind and if you keep 558 00:27:08,610 --> 00:27:12,060 in mind and think about what Sara said earlier about the 559 00:27:12,060 --> 00:27:14,470 data that you decide to collect, one problem with 560 00:27:14,470 --> 00:27:16,850 board games is with all the interesting conversations 561 00:27:16,850 --> 00:27:19,960 happening around the table and multiple players suggesting 562 00:27:19,960 --> 00:27:22,992 strategies to each other, you can come up with reams of 563 00:27:22,992 --> 00:27:24,390 notes in a single play test session. 564 00:27:24,390 --> 00:27:29,180 How do you decide to digest that is really important. 565 00:27:29,180 --> 00:27:33,735 And you're the only person there and you're the only 566 00:27:33,735 --> 00:27:35,611 person on your team, you need to try to let the rest 567 00:27:35,611 --> 00:27:36,840 of your team know. 568 00:27:36,840 --> 00:27:39,050 You need to put in the work of digesting all of that 569 00:27:39,050 --> 00:27:43,860 information before necessarily giving it over to your team. 570 00:27:43,860 --> 00:27:47,780 SARA VERRILLI: One possible solution also is you can get 571 00:27:47,780 --> 00:27:52,680 permission from play testers for a tape recorder. 572 00:27:52,680 --> 00:27:56,476 Just get a digital recorder and record all of the 573 00:27:56,476 --> 00:27:59,060 conversation as it goes on. 574 00:27:59,060 --> 00:28:01,596 PROFESSOR: But you still want to pick out the keys, you 575 00:28:01,596 --> 00:28:04,090 don't want to revisit the entire conversation. 576 00:28:04,090 --> 00:28:07,873 Because if the game took an hour, and every single one of 577 00:28:07,873 --> 00:28:11,240 your team members visits, by now you've lost three hours. 578 00:28:11,240 --> 00:28:14,350 So you want to find that, do that little bit of work to 579 00:28:14,350 --> 00:28:16,580 save yourself the detail. 580 00:28:16,580 --> 00:28:19,570 But that's the way because you can focus, you can take down 581 00:28:19,570 --> 00:28:23,110 notes on the main points and just let the tape recorder 582 00:28:23,110 --> 00:28:26,370 record all the details so that you don't have to worry about 583 00:28:26,370 --> 00:28:29,295 recording and really, the main points are the things that you 584 00:28:29,295 --> 00:28:30,130 want to make sure you get across to your team. 585 00:28:30,130 --> 00:28:32,526 The tape recorder is just evidence. 586 00:28:32,526 --> 00:28:37,460 You could use video depending on who you're recording with. 587 00:28:37,460 --> 00:28:40,604 Sometimes people get kind of antsy when they 588 00:28:40,604 --> 00:28:41,880 know there's a camera. 589 00:28:41,880 --> 00:28:43,109 That being said-- 590 00:28:43,109 --> 00:28:48,099 SARA VERRILLI: And also, just put the video tilted at the 591 00:28:48,099 --> 00:28:48,941 board so you're not recording the players, you're 592 00:28:48,941 --> 00:28:49,097 recording the game. 593 00:28:49,097 --> 00:28:50,842 PROFESSOR: Actually webcams are really good for 594 00:28:50,842 --> 00:28:51,093 that sort of thing. 595 00:28:51,093 --> 00:28:51,592 SARA VERRILLI: Yeah. 596 00:28:51,592 --> 00:28:55,782 PROFESSOR: You don't run out of tape, just record straight 597 00:28:55,782 --> 00:28:56,083 to hard drive. 598 00:28:56,083 --> 00:28:56,582 SARA VERRILLI: Easy to fast forward through, also. 599 00:28:56,582 --> 00:29:00,740 PROFESSOR: Yeah, with your editing tools. 600 00:29:00,740 --> 00:29:02,560 Let's see. 601 00:29:02,560 --> 00:29:05,820 So when you're reporting to other people, think like 602 00:29:05,820 --> 00:29:08,640 bullet points for instance. 603 00:29:08,640 --> 00:29:13,410 Huge long paragraphs of prose, not so much. 604 00:29:13,410 --> 00:29:14,490 Let's see. 605 00:29:14,490 --> 00:29:16,930 Try to avoid using the following words, broken, 606 00:29:16,930 --> 00:29:25,000 problem, unusable, confusing, kind of odd, you can get more 607 00:29:25,000 --> 00:29:27,680 specific than that, but these are words that kind of like 608 00:29:27,680 --> 00:29:31,150 imply not only is there a problem, someone made a 609 00:29:31,150 --> 00:29:34,900 mistake in his design of the game. 610 00:29:34,900 --> 00:29:39,589 And truth be told, most of the time it wasn't a mistake, you 611 00:29:39,589 --> 00:29:41,923 just didn't have the complete play testing, you probably 612 00:29:41,923 --> 00:29:42,740 just had the designer. 613 00:29:42,740 --> 00:29:44,575 But you don't want to make someone feel like they've made 614 00:29:44,575 --> 00:29:49,060 a mistake because that makes them defensive and then 615 00:29:49,060 --> 00:29:51,814 they're going to try to defend how what they wrote in the 616 00:29:51,814 --> 00:29:55,230 first place is actually correct, which is actually not 617 00:29:55,230 --> 00:29:56,480 correct either. 618 00:29:58,850 --> 00:30:01,934 You can talk about features that need attention. 619 00:30:01,934 --> 00:30:06,774 With attention, this feature could be confused. 620 00:30:06,774 --> 00:30:10,270 I guess that's actually on the edge there. 621 00:30:10,270 --> 00:30:18,722 So but just generally, pick your words carefully and one 622 00:30:18,722 --> 00:30:23,024 thing that Sara mentioned earlier was that testers are 623 00:30:23,024 --> 00:30:26,080 not designers and key designers are actually not a 624 00:30:26,080 --> 00:30:27,397 good testing team. 625 00:30:27,397 --> 00:30:31,728 So for the most part, you are immediately going to exclude 626 00:30:31,728 --> 00:30:34,737 yourself from your target audience because you know too 627 00:30:34,737 --> 00:30:35,010 much of the game. 628 00:30:35,010 --> 00:30:36,590 This is also covered in your book. 629 00:30:36,590 --> 00:30:39,210 As soon as you've played through your game once or 630 00:30:39,210 --> 00:30:40,290 twice, that's it. 631 00:30:40,290 --> 00:30:41,800 Your kind of work is finished. 632 00:30:41,800 --> 00:30:46,626 You can still be a good technical tester, I guess. 633 00:30:46,626 --> 00:30:49,914 You can still check whether some mechanics are working the 634 00:30:49,914 --> 00:30:53,240 way that you want to work, but you can't tell whether the 635 00:30:53,240 --> 00:30:55,232 dynamics in your settings are working anymore. 636 00:30:55,232 --> 00:30:58,718 You have to get your focus testing done. 637 00:30:58,718 --> 00:31:02,204 SARA VERRILLI: You also really can't tell, you can never 638 00:31:02,204 --> 00:31:05,690 tell-- you can never share your usability tester. 639 00:31:05,690 --> 00:31:07,184 It will always make sense to you. 640 00:31:07,184 --> 00:31:10,504 It's amazing that what makes sense to you is completely 641 00:31:10,504 --> 00:31:14,156 incomprehensible to the very intelligent person you give 642 00:31:14,156 --> 00:31:15,152 the game to. 643 00:31:15,152 --> 00:31:16,397 Really, truly, I've been thinking 644 00:31:16,397 --> 00:31:17,144 about this one myself. 645 00:31:17,144 --> 00:31:22,124 PROFESSOR: Usability testing, by the way, is one of those 646 00:31:22,124 --> 00:31:24,655 things that even if you only have the chance to do a pop-up 647 00:31:24,655 --> 00:31:29,170 partial test like, all I have is this one card, the game's 648 00:31:29,170 --> 00:31:34,130 not complete, but grab someone walking down the corridor on 649 00:31:34,130 --> 00:31:37,939 the way to the washroom and say, come in here and take a 650 00:31:37,939 --> 00:31:38,980 look at this thing I'm working on. 651 00:31:38,980 --> 00:31:39,950 Even that is better information. 652 00:31:39,950 --> 00:31:42,990 STUDENT: Hey kid, come in here. 653 00:31:42,990 --> 00:31:46,120 Come in here, I have something to show you? 654 00:31:46,120 --> 00:31:46,590 I have a surprise for you. 655 00:31:46,590 --> 00:31:47,530 [LAUGHTER] 656 00:31:47,530 --> 00:31:51,220 SARA VERRILLI: Actually, you can get going testing for 657 00:31:51,220 --> 00:31:52,090 [? serious. ?] 658 00:31:52,090 --> 00:31:54,175 STUDENT: Free candy. 659 00:31:54,175 --> 00:31:56,676 PROFESSOR: [? Mona ?] actually calls it employee kidnapping. 660 00:31:56,676 --> 00:31:59,870 [LAUGHTER] 661 00:31:59,870 --> 00:32:07,400 Of course, he's the creative director, so he can do that. 662 00:32:07,400 --> 00:32:10,786 SARA VERRILLI: It works best if you're accosting people 663 00:32:10,786 --> 00:32:13,370 whom you have power over. 664 00:32:13,370 --> 00:32:14,610 PROFESSOR: The audio thing, by the way, is one of those 665 00:32:14,610 --> 00:32:17,730 things that could be really useful if people are difficult 666 00:32:17,730 --> 00:32:18,580 [? pundits ?]. 667 00:32:18,580 --> 00:32:22,415 And so it might be one of those things that you want to 668 00:32:22,415 --> 00:32:25,880 keep in reserve, it's like, OK, there is a problem here 669 00:32:25,880 --> 00:32:28,005 and the team goes, I don't think there's a problem here. 670 00:32:28,005 --> 00:32:30,280 I don't understand why anyone's having problems here. 671 00:32:30,280 --> 00:32:35,522 Then you quit, because it's kind of a bludgeon 672 00:32:35,522 --> 00:32:37,560 actually on the team. 673 00:32:37,560 --> 00:32:40,410 On one hand, if you have a good agent team, they're going 674 00:32:40,410 --> 00:32:43,400 hear it and go, oh my god, we have a problem and immediately 675 00:32:43,400 --> 00:32:44,417 try to fix it. 676 00:32:44,417 --> 00:32:45,770 But it's awesome. 677 00:32:45,770 --> 00:32:50,868 On the other hand, it's like, well, do you need to do a 678 00:32:50,868 --> 00:32:54,510 recording for every single offer that we have? 679 00:32:54,510 --> 00:32:55,730 That's something to be known about, so 680 00:32:55,730 --> 00:32:57,456 keep that in reserve. 681 00:33:01,210 --> 00:33:03,010 I think that's pretty much it. 682 00:33:09,310 --> 00:33:12,610 I have an exercise, but I think [UNINTELLIGIBLE] 683 00:33:12,610 --> 00:33:14,410 so I can turn that off. 684 00:33:18,010 --> 00:33:19,810 Do you have any questions while I 685 00:33:19,810 --> 00:33:21,310 graph the data segments? 686 00:33:21,310 --> 00:33:21,610 No? 687 00:33:21,610 --> 00:33:22,860 OK.