1 00:00:00,060 --> 00:00:02,500 The following content is provided under a Creative 2 00:00:02,500 --> 00:00:04,019 Commons license. 3 00:00:04,019 --> 00:00:06,360 Your support will help MIT OpenCourseWare 4 00:00:06,360 --> 00:00:10,730 continue to offer high quality educational resources for free. 5 00:00:10,730 --> 00:00:13,330 To make a donation or view additional materials 6 00:00:13,330 --> 00:00:17,215 from 100 of MIT courses, visit MIT OpenCourseWare 7 00:00:17,215 --> 00:00:17,840 at ocw.mit.edu. 8 00:00:21,360 --> 00:00:22,594 RICHARD EBERHARDT: Hi, Pablo. 9 00:00:22,594 --> 00:00:23,350 PABLO SUAREZ: Good to be back. 10 00:00:23,350 --> 00:00:25,016 RICHARD EBERHARDT: Nice having you here. 11 00:00:25,016 --> 00:00:28,880 So we wanted to talk about a number 12 00:00:28,880 --> 00:00:32,940 of high-level postmortemy things for how the class went. 13 00:00:32,940 --> 00:00:36,110 So we've got some high-level things we wanted to talk about. 14 00:00:36,110 --> 00:00:39,160 We want to talk about the collaboration between MIT 15 00:00:39,160 --> 00:00:42,790 and the Red Cross/Red Crescent group and your folks-- 16 00:00:42,790 --> 00:00:45,960 how student preparedness went, in particular, comparing 17 00:00:45,960 --> 00:00:49,750 those teams that got a lot of support, like client support, 18 00:00:49,750 --> 00:00:50,687 like Snap, of course. 19 00:00:50,687 --> 00:00:53,020 But also, some of the other teams got a little bit more, 20 00:00:53,020 --> 00:00:54,690 and then the unsupported teams, which 21 00:00:54,690 --> 00:00:57,250 aren't exactly unsupported, but they got less, 22 00:00:57,250 --> 00:01:00,250 so that's largely the Heatwave Team, 23 00:01:00,250 --> 00:01:03,350 I think was the main one that we want to talk about there. 24 00:01:03,350 --> 00:01:07,770 So collaboration what do you all think about how 25 00:01:07,770 --> 00:01:08,934 we talked to each other? 26 00:01:08,934 --> 00:01:10,850 How we met with each other-- things like that? 27 00:01:13,755 --> 00:01:15,130 PHILIP TAN: I like the way how we 28 00:01:15,130 --> 00:01:17,150 started in the beginning of the semester. 29 00:01:17,150 --> 00:01:20,280 We had to talk-- and we had some Skype issues. 30 00:01:20,280 --> 00:01:23,470 But I think the idea of kicking off the entire class 31 00:01:23,470 --> 00:01:27,210 with the initial presentation of this 32 00:01:27,210 --> 00:01:28,960 is what your final project is going to be. 33 00:01:28,960 --> 00:01:30,751 It's going to take us a while to get there, 34 00:01:30,751 --> 00:01:32,620 but you come into class understanding 35 00:01:32,620 --> 00:01:33,870 this is what it's going to be. 36 00:01:33,870 --> 00:01:38,074 And Pablo being the person to describe 37 00:01:38,074 --> 00:01:39,490 why this is going to be important, 38 00:01:39,490 --> 00:01:42,690 I think, set lot of expectations well. 39 00:01:42,690 --> 00:01:51,310 So I think later on, when we actually got to that project, 40 00:01:51,310 --> 00:01:53,590 the degree to which each team actually 41 00:01:53,590 --> 00:01:55,770 could have been supported actually 42 00:01:55,770 --> 00:01:58,256 became more of an issue. 43 00:01:58,256 --> 00:02:00,130 The teams that you could directly talk with-- 44 00:02:00,130 --> 00:02:04,460 and you were going to be the primary user of the game-- 45 00:02:04,460 --> 00:02:07,456 clearly benefited. 46 00:02:07,456 --> 00:02:09,289 But you were right there from the beginning. 47 00:02:09,289 --> 00:02:13,050 These are the teams that, maybe, they did receive support, 48 00:02:13,050 --> 00:02:15,570 or some of them didn't receive support, 49 00:02:15,570 --> 00:02:17,420 but weren't talking with people who 50 00:02:17,420 --> 00:02:23,366 had been there right from the start, I think maybe 51 00:02:23,366 --> 00:02:24,990 had felt a little bit more disconnected 52 00:02:24,990 --> 00:02:28,480 from the main thrust of the class. 53 00:02:28,480 --> 00:02:31,190 At least when they got to that final assignment. 54 00:02:31,190 --> 00:02:35,190 So something I like to think about for future classes 55 00:02:35,190 --> 00:02:38,080 is the extent to which-- if we're 56 00:02:38,080 --> 00:02:41,030 going to repeat this format-- where the primary client comes 57 00:02:41,030 --> 00:02:44,870 in right at the beginning to set the tone for the entire class. 58 00:02:44,870 --> 00:02:49,120 How do we represent more of the variety of different things 59 00:02:49,120 --> 00:02:50,745 that you can do right at the beginning? 60 00:02:53,770 --> 00:02:55,760 So that when students think about what they're 61 00:02:55,760 --> 00:02:59,590 going to be doing throughout the entire semester, 62 00:02:59,590 --> 00:03:03,350 they are actually keeping full scope of the project in mind, 63 00:03:03,350 --> 00:03:06,400 not just the one example that might be presented to them 64 00:03:06,400 --> 00:03:07,544 on the first day of class. 65 00:03:07,544 --> 00:03:08,960 PABLO SUAREZ: And we also may want 66 00:03:08,960 --> 00:03:12,170 to have something like an engagement meter. 67 00:03:12,170 --> 00:03:14,330 Because I wasn't aware of the fact 68 00:03:14,330 --> 00:03:17,420 that some teams were feeling disengaged. 69 00:03:17,420 --> 00:03:19,430 I would show up, some people have questions, 70 00:03:19,430 --> 00:03:21,440 some other have less questions. 71 00:03:21,440 --> 00:03:25,550 And from my perspective, those who have questions I spent time 72 00:03:25,550 --> 00:03:29,820 and it was about the same chronometric time per group. 73 00:03:29,820 --> 00:03:31,540 I could see that some were more lost, 74 00:03:31,540 --> 00:03:33,262 but I wasn't aware that what was needed 75 00:03:33,262 --> 00:03:34,470 was more fire in their belly. 76 00:03:34,470 --> 00:03:35,240 So if we could-- 77 00:03:35,240 --> 00:03:35,960 RICHARD EBERHARDT: I couldn't see that either. 78 00:03:35,960 --> 00:03:36,350 PABLO SUAREZ: Right. 79 00:03:36,350 --> 00:03:38,683 RICHARD EBERHARDT: The students never reported it to us. 80 00:03:38,683 --> 00:03:40,480 SARA VERRILLI: Yeah, I was going to say, 81 00:03:40,480 --> 00:03:42,230 when we asked the students to do their postmortems, 82 00:03:42,230 --> 00:03:45,070 they were really clear about how this had been a big motivation, 83 00:03:45,070 --> 00:03:47,140 and interest had been a big problem for them. 84 00:03:47,140 --> 00:03:50,170 But they never mentioned it to us during check-in meetings 85 00:03:50,170 --> 00:03:53,150 when we were doing play testing, when we were working with them. 86 00:03:53,150 --> 00:03:55,420 So that's the sort of thing that we 87 00:03:55,420 --> 00:03:57,420 need to-- probably one of the things 88 00:03:57,420 --> 00:04:00,920 you do is probably to warn them about that. 89 00:04:00,920 --> 00:04:03,570 These are hard topics, and they can become boring topics 90 00:04:03,570 --> 00:04:05,850 very quickly. 91 00:04:05,850 --> 00:04:10,010 And they need to step up and let us know when they think 92 00:04:10,010 --> 00:04:12,020 they're facing a brick wall. 93 00:04:12,020 --> 00:04:14,960 RICHARD EBERHARDT: I think the clues were there. 94 00:04:14,960 --> 00:04:18,649 If we had means and a way to remind ourselves, 95 00:04:18,649 --> 00:04:21,160 like a checklist or something simple like that, just to say, 96 00:04:21,160 --> 00:04:22,210 are you engaged? 97 00:04:22,210 --> 00:04:23,604 What's your motivation? 98 00:04:23,604 --> 00:04:25,270 And, again, that's assuming that we even 99 00:04:25,270 --> 00:04:27,070 want that to be our job. 100 00:04:27,070 --> 00:04:28,540 I think for anything client based, 101 00:04:28,540 --> 00:04:30,037 it has be our, probably, our job. 102 00:04:30,037 --> 00:04:31,620 SARA VERRILLI: I think that when we're 103 00:04:31,620 --> 00:04:35,250 asking them to work on projects where they have relatively 104 00:04:35,250 --> 00:04:38,587 little control over the topic, I think we actually do need to. 105 00:04:38,587 --> 00:04:40,170 Because I think part of it was I don't 106 00:04:40,170 --> 00:04:42,050 think the teams knew how much freedom they 107 00:04:42,050 --> 00:04:46,622 had to deviate from the topic they had chosen 108 00:04:46,622 --> 00:04:47,330 and the specific. 109 00:04:47,330 --> 00:04:48,996 They thought they had chosen this thing, 110 00:04:48,996 --> 00:04:50,900 and now they were stuck with it. 111 00:04:50,900 --> 00:04:53,180 Even though one team was like, we 112 00:04:53,180 --> 00:04:55,060 chose a topic that just wasn't deep enough. 113 00:04:55,060 --> 00:04:57,490 There wasn't enough there. 114 00:04:57,490 --> 00:04:59,320 And if they had come and talked to us, 115 00:04:59,320 --> 00:05:00,820 we could have given them the freedom 116 00:05:00,820 --> 00:05:03,110 to well, go change to a new topic, or how can we go-- 117 00:05:03,110 --> 00:05:03,330 RICHARD EBERHARDT: Go deeper. 118 00:05:03,330 --> 00:05:04,600 SARA VERRILLI: --how can we go deeper in it? 119 00:05:04,600 --> 00:05:05,340 Right. 120 00:05:05,340 --> 00:05:08,010 But we were not sufficiently warned. 121 00:05:08,010 --> 00:05:09,840 PHILIP TAN: But that's where rapport 122 00:05:09,840 --> 00:05:13,450 with someone who knows a lot about the domain can help. 123 00:05:13,450 --> 00:05:16,720 And if they don't feel like they have that rapport or someone 124 00:05:16,720 --> 00:05:20,140 who they can readily contact to ask even seemingly 125 00:05:20,140 --> 00:05:22,522 silly questions, then we're not going 126 00:05:22,522 --> 00:05:24,105 to be able to get down to actually ask 127 00:05:24,105 --> 00:05:25,270 the important questions. 128 00:05:25,270 --> 00:05:26,686 Because sometimes they can't tell. 129 00:05:26,686 --> 00:05:30,190 They can't necessarily tell what looks like flagging motivation 130 00:05:30,190 --> 00:05:32,600 on a part of the team or what just 131 00:05:32,600 --> 00:05:35,911 seems like end-of-semester pressures on the students, 132 00:05:35,911 --> 00:05:36,410 right? 133 00:05:39,480 --> 00:05:41,460 They haven't been working on enough projects 134 00:05:41,460 --> 00:05:43,690 to be able to tell the difference. 135 00:05:43,690 --> 00:05:46,490 And so we do need to help them through that, I think. 136 00:05:46,490 --> 00:05:47,870 SARA VERRILLI: By that you mean the students, then? 137 00:05:47,870 --> 00:05:48,575 PHILIP TAN: The students. 138 00:05:48,575 --> 00:05:48,920 I mean the students. 139 00:05:48,920 --> 00:05:51,420 PABLO SUAREZ: And one of the things you may want to consider 140 00:05:51,420 --> 00:05:53,950 is we know that both in the real life-- in the future, 141 00:05:53,950 --> 00:05:56,800 in their future, as well as in future courses-- 142 00:05:56,800 --> 00:05:58,950 there will be limited ability of those 143 00:05:58,950 --> 00:06:01,600 who are experts to interact with the game design team. 144 00:06:01,600 --> 00:06:03,781 The game design team, you took the effort-- 145 00:06:03,781 --> 00:06:06,030 and I thought it was very well-balanced-- to have some 146 00:06:06,030 --> 00:06:08,640 who know programming, some who know narrative, some who 147 00:06:08,640 --> 00:06:11,340 know user interface, and so on. 148 00:06:11,340 --> 00:06:13,480 They will, by default, not have expertise 149 00:06:13,480 --> 00:06:17,780 on heatwaves or cholera or etc or climate science. 150 00:06:17,780 --> 00:06:21,740 They cannot assume that all they need to know-- 151 00:06:21,740 --> 00:06:23,650 including the appetite for engagement-- 152 00:06:23,650 --> 00:06:26,730 will be only in the client. 153 00:06:26,730 --> 00:06:32,530 MIT is a bubbling powerhouse of engaging stuff. 154 00:06:32,530 --> 00:06:35,825 And if they could, as part of what they are told-- they 155 00:06:35,825 --> 00:06:37,450 will have to do in the future, and what 156 00:06:37,450 --> 00:06:39,620 they have to do in this course, to reach out 157 00:06:39,620 --> 00:06:44,350 to engaging people, fellow students, and junior faculty 158 00:06:44,350 --> 00:06:48,430 or whoever may give them some time-- to just go and say, 159 00:06:48,430 --> 00:06:49,910 what is exciting about this? 160 00:06:49,910 --> 00:06:51,500 What are the cutting-edge areas? 161 00:06:51,500 --> 00:06:53,120 What are the institutional challenges. 162 00:06:53,120 --> 00:06:56,355 So that they also have to fuel their own appetite. 163 00:06:56,355 --> 00:06:57,980 RICHARD EBERHARDT: Having an assignment 164 00:06:57,980 --> 00:07:01,000 or having part of that final project being like here's 165 00:07:01,000 --> 00:07:03,650 the research component that we expect you to do, 166 00:07:03,650 --> 00:07:06,076 but we didn't put an underline over it. 167 00:07:06,076 --> 00:07:08,700 SARA VERRILLI: We didn't make it explicit that they needed to-- 168 00:07:08,700 --> 00:07:10,741 RICHARD EBERHARDT: We did when it was in trouble. 169 00:07:10,741 --> 00:07:14,080 Like for Saving Gora Gora, they were in big trouble. 170 00:07:14,080 --> 00:07:16,750 We told them a number-- you told them, we told them. 171 00:07:16,750 --> 00:07:19,220 I think you'd be surprised what they came up with. 172 00:07:19,220 --> 00:07:21,672 Because they actually, I think, they took the baton 173 00:07:21,672 --> 00:07:22,380 and went with it. 174 00:07:22,380 --> 00:07:23,190 SARA VERRILLI: They did that [INAUDIBLE]. 175 00:07:23,190 --> 00:07:23,490 PHILIP TAN: Nice. 176 00:07:23,490 --> 00:07:25,609 RICHARD EBERHARDT: The other cholera team did not, 177 00:07:25,609 --> 00:07:27,150 and they were given the same-- almost 178 00:07:27,150 --> 00:07:28,580 the exact same-- feedback. 179 00:07:28,580 --> 00:07:29,330 That is the thing. 180 00:07:29,330 --> 00:07:31,970 PHILIP TAN: I think there is a huge social barrier 181 00:07:31,970 --> 00:07:36,360 between the undergraduates and the rest of the institute. 182 00:07:36,360 --> 00:07:39,620 The noble laureates, and even the post-docs here, 183 00:07:39,620 --> 00:07:44,070 that's a lot of social pressure keeping students away 184 00:07:44,070 --> 00:07:45,650 from that kind of interaction. 185 00:07:45,650 --> 00:07:46,660 You're all right. 186 00:07:46,660 --> 00:07:50,086 All of these things do exist here at MIT, but some of this 187 00:07:50,086 --> 00:07:51,970 is reinforced by professors themselves-- 188 00:07:51,970 --> 00:07:53,553 professors who keep their doors closed 189 00:07:53,553 --> 00:07:56,120 when they in their office, things like that. 190 00:07:56,120 --> 00:08:01,680 And so I do think we have to do give them the license, 191 00:08:01,680 --> 00:08:04,014 and sometimes that's pointing them to specific people. 192 00:08:04,014 --> 00:08:06,430 RICHARD EBERHARDT: Yeah, giving them resources like here's 193 00:08:06,430 --> 00:08:07,220 a list. 194 00:08:07,220 --> 00:08:08,710 One of these might pan out for you. 195 00:08:08,710 --> 00:08:09,585 The others might not. 196 00:08:09,585 --> 00:08:13,950 And that's OK, just do what you can, kind of prime them for it. 197 00:08:13,950 --> 00:08:17,724 PHILIP TAN: It's hard to be a teenager in this space. 198 00:08:17,724 --> 00:08:19,390 RICHARD EBERHARDT: Actually, since we're 199 00:08:19,390 --> 00:08:20,973 talking about the student preparedness 200 00:08:20,973 --> 00:08:24,880 in that research like what we saw between what we consider 201 00:08:24,880 --> 00:08:28,152 the supported teams versus the unsupported teams, which 202 00:08:28,152 --> 00:08:30,110 may or may not be fair-- could be the motivated 203 00:08:30,110 --> 00:08:32,150 teams versus the unmotivated teams. 204 00:08:32,150 --> 00:08:33,826 Snap, in particular, really-- 205 00:08:33,826 --> 00:08:35,409 PABLO SUAREZ: [INAUDIBLE] correlation. 206 00:08:35,409 --> 00:08:37,809 RICHARD EBERHARDT: --really well-supported. 207 00:08:37,809 --> 00:08:41,977 They actually reached out to you all for help with the game. 208 00:08:41,977 --> 00:08:44,310 You actually even did testing for them that was actually 209 00:08:44,310 --> 00:08:46,780 coming from you and Jana, yeah, because you 210 00:08:46,780 --> 00:08:47,800 were running workshops. 211 00:08:47,800 --> 00:08:48,633 PABLO SUAREZ: Right. 212 00:08:48,633 --> 00:08:51,140 RICHARD EBERHARDT: How did those workshops run-- how did you 213 00:08:51,140 --> 00:08:52,980 interact with the team? 214 00:08:52,980 --> 00:08:54,570 PABLO SUAREZ: So, first of all, I 215 00:08:54,570 --> 00:08:58,070 think it's important to distinguish not only what you 216 00:08:58,070 --> 00:08:59,990 are calling supportive versus unsupportive, 217 00:08:59,990 --> 00:09:01,950 but also Snap versus everything else. 218 00:09:01,950 --> 00:09:04,660 Because Snap was a game concept that 219 00:09:04,660 --> 00:09:07,750 preexisted the course that needed refinement 220 00:09:07,750 --> 00:09:09,815 and digital interface. 221 00:09:09,815 --> 00:09:11,690 So from that perspective, it's not surprising 222 00:09:11,690 --> 00:09:13,150 that they have something clearly defined that they 223 00:09:13,150 --> 00:09:14,710 could run faster and deeper. 224 00:09:14,710 --> 00:09:17,040 And the group was a group that engaged and went 225 00:09:17,040 --> 00:09:19,890 deep and went fast. 226 00:09:19,890 --> 00:09:21,970 The interaction we had with them, 227 00:09:21,970 --> 00:09:25,450 also, very importantly, we have events 228 00:09:25,450 --> 00:09:27,250 that we could use the product. 229 00:09:27,250 --> 00:09:28,930 So it was much easier to give feedback 230 00:09:28,930 --> 00:09:30,445 saying when we do this, we expect 231 00:09:30,445 --> 00:09:33,070 people that would get bored, why don't we try this other thing. 232 00:09:33,070 --> 00:09:34,528 RICHARD EBERHARDT: It's milestones, 233 00:09:34,528 --> 00:09:35,950 which we give them milestones. 234 00:09:35,950 --> 00:09:37,630 But there is no weight behind them. 235 00:09:37,630 --> 00:09:40,620 These ones have an actual, if it doesn't work by then, 236 00:09:40,620 --> 00:09:42,670 someone's going to have egg on their face, right? 237 00:09:42,670 --> 00:09:45,000 PABLO SUAREZ: And from that perspective, we know this, 238 00:09:45,000 --> 00:09:49,820 but we can try to do more of giving students 239 00:09:49,820 --> 00:09:53,170 more proportion of work that we know-- that we, the clients 240 00:09:53,170 --> 00:09:54,800 know-- that we have milestones. 241 00:09:54,800 --> 00:09:57,570 And that is limited by how much of the real world 242 00:09:57,570 --> 00:09:59,630 gives us milestones to abide by. 243 00:09:59,630 --> 00:10:01,310 But we can try that. 244 00:10:01,310 --> 00:10:05,360 I do not feel comfortable with the label supportive versus 245 00:10:05,360 --> 00:10:10,150 unsupportive because it sounds like some got time and support 246 00:10:10,150 --> 00:10:11,660 and some others didn't. 247 00:10:11,660 --> 00:10:15,520 I think it's more about the form of the relationship 248 00:10:15,520 --> 00:10:18,640 and how the time was used or the opportunity to use the time 249 00:10:18,640 --> 00:10:19,510 was used. 250 00:10:19,510 --> 00:10:22,990 Because I have no doubt that you supported the team when 251 00:10:22,990 --> 00:10:25,110 you knew support was needed. 252 00:10:25,110 --> 00:10:28,280 And as far as I know, the same was true on our part. 253 00:10:28,280 --> 00:10:30,520 I think it's more about having sensors 254 00:10:30,520 --> 00:10:33,340 about when support is needed that is not being delivered. 255 00:10:33,340 --> 00:10:35,673 RICHARD EBERHARDT: I think when we're talking about them 256 00:10:35,673 --> 00:10:38,150 in this way, I'm using it based on how 257 00:10:38,150 --> 00:10:40,050 they describe their own relationship 258 00:10:40,050 --> 00:10:41,480 and their own experience. 259 00:10:41,480 --> 00:10:43,515 So that the teams who we are calling unsupported 260 00:10:43,515 --> 00:10:45,140 are the ones who, in their postmortems, 261 00:10:45,140 --> 00:10:46,598 called out and said, we didn't feel 262 00:10:46,598 --> 00:10:48,240 like we got client support. 263 00:10:48,240 --> 00:10:49,660 They got the same amount. 264 00:10:49,660 --> 00:10:52,000 It was a different way, different quality, 265 00:10:52,000 --> 00:10:53,110 different quantity. 266 00:10:53,110 --> 00:10:55,693 SARA VERRILLI: The way to put it is the same amount of support 267 00:10:55,693 --> 00:10:58,199 was there for them. 268 00:10:58,199 --> 00:10:59,990 One other way to put it is they didn't know 269 00:10:59,990 --> 00:11:02,590 how to reach out and get it. 270 00:11:02,590 --> 00:11:04,170 They didn't use the resources that 271 00:11:04,170 --> 00:11:06,810 were there because, at some level, 272 00:11:06,810 --> 00:11:08,470 we weren't pushing them at them. 273 00:11:08,470 --> 00:11:11,950 And a lot of MIT courses tend to encourage students to solve 274 00:11:11,950 --> 00:11:13,454 all their problems themselves. 275 00:11:13,454 --> 00:11:14,870 They don't want them reaching out. 276 00:11:14,870 --> 00:11:17,430 They want them-- figure this out. 277 00:11:17,430 --> 00:11:20,315 And so it may come as more of a second-- 278 00:11:20,315 --> 00:11:22,440 it may not come to them as first nature to go, hey, 279 00:11:22,440 --> 00:11:25,065 we need to go and reach out for more help and grab that-- 280 00:11:25,065 --> 00:11:27,440 RICHARD EBERHARDT: And so that's actually the same thing. 281 00:11:27,440 --> 00:11:29,020 So what we saw with the Snap team is 282 00:11:29,020 --> 00:11:31,360 we wanted-- the feedback they were getting from us 283 00:11:31,360 --> 00:11:33,615 was look at your front end, look at your UI. 284 00:11:33,615 --> 00:11:37,370 We wanted to see a lot of changes on the UI. 285 00:11:37,370 --> 00:11:39,365 Because they were working on these milestones, 286 00:11:39,365 --> 00:11:41,250 they were really worried about the back end 287 00:11:41,250 --> 00:11:43,291 so they were doing a lot of work in the back end. 288 00:11:46,272 --> 00:11:47,980 They were getting two different messages, 289 00:11:47,980 --> 00:11:50,970 and they chose client message rather than instructor message. 290 00:11:50,970 --> 00:11:52,275 Totally OK and totally viable. 291 00:11:54,997 --> 00:11:56,080 We took that into account. 292 00:11:56,080 --> 00:11:58,260 But I think the main issues we had 293 00:11:58,260 --> 00:12:00,150 with them is when they got our message, 294 00:12:00,150 --> 00:12:02,317 it was, they heard it early on. 295 00:12:02,317 --> 00:12:04,650 They didn't react to it until it was too late-- had they 296 00:12:04,650 --> 00:12:07,220 done a little bit extra in the beginning. 297 00:12:07,220 --> 00:12:09,770 So one thing I think we need to think about with the class 298 00:12:09,770 --> 00:12:13,410 is thinking about how are we helping them understand 299 00:12:13,410 --> 00:12:16,570 how to prioritize feedback? 300 00:12:16,570 --> 00:12:17,880 Right? 301 00:12:17,880 --> 00:12:20,500 What's more important? 302 00:12:20,500 --> 00:12:22,350 Could be because here's something that's 303 00:12:22,350 --> 00:12:23,474 actually going to get used. 304 00:12:23,474 --> 00:12:26,230 It could also be here's a lesson we want to make sure they get. 305 00:12:26,230 --> 00:12:27,396 I think they got the lesson. 306 00:12:27,396 --> 00:12:29,314 They just got it in a harder way. 307 00:12:29,314 --> 00:12:31,730 PHILIP TAN: Sometimes they get a lesson at a point of time 308 00:12:31,730 --> 00:12:33,880 that they can't actually execute on it. 309 00:12:33,880 --> 00:12:37,120 But hopefully they can do it for their next project after class. 310 00:12:39,650 --> 00:12:41,580 Something I want to suggest, actually, 311 00:12:41,580 --> 00:12:44,120 because we've been talking about how it's been difficult 312 00:12:44,120 --> 00:12:48,040 for us to figure out what's going on inside the teams. 313 00:12:48,040 --> 00:12:51,410 We can see the games and the testing, 314 00:12:51,410 --> 00:12:55,100 but the team dynamics, we don't find out about that 315 00:12:55,100 --> 00:12:57,950 until near the end of the class. 316 00:12:57,950 --> 00:13:02,010 And something that we used to do outside of this class, 317 00:13:02,010 --> 00:13:04,640 but in our research work, was to get all off 318 00:13:04,640 --> 00:13:07,540 the scrum masters together on a regular basis 319 00:13:07,540 --> 00:13:11,497 to just do an exchange of information with each other. 320 00:13:11,497 --> 00:13:13,830 They don't even have to be necessarily reporting to us-- 321 00:13:13,830 --> 00:13:16,020 although we can moderate those sessions. 322 00:13:16,020 --> 00:13:19,270 It's more about them sharing their experiences with each 323 00:13:19,270 --> 00:13:22,970 other, and then it may make it more obvious that there are 324 00:13:22,970 --> 00:13:25,849 resources out there that some teams are taking advantage 325 00:13:25,849 --> 00:13:26,890 of and some teams aren't. 326 00:13:30,094 --> 00:13:32,260 Of course, we can use those situations for teaching, 327 00:13:32,260 --> 00:13:37,640 but also for teaching each other. 328 00:13:37,640 --> 00:13:40,225 SARA VERRILLI: Creating some group, group support. 329 00:13:40,225 --> 00:13:41,850 RICHARD EBERHARDT: We talked about that 330 00:13:41,850 --> 00:13:44,590 before this semester started, but we couldn't figure out 331 00:13:44,590 --> 00:13:47,860 a way to do that fairly. 332 00:13:47,860 --> 00:13:49,900 I already feel producers bear the brunt 333 00:13:49,900 --> 00:13:52,419 of the work in the class. 334 00:13:52,419 --> 00:13:53,960 They're the ones we see all the time. 335 00:13:53,960 --> 00:13:56,480 We don't see the students working on programming 336 00:13:56,480 --> 00:13:57,530 or design as much. 337 00:13:57,530 --> 00:13:58,980 Design a little bit more, but programming, 338 00:13:58,980 --> 00:14:00,230 we just almost never see them. 339 00:14:00,230 --> 00:14:03,230 SARA VERRILLI: And we let the teams use the producers 340 00:14:03,230 --> 00:14:06,110 as their face-- so giving all the presentations, 341 00:14:06,110 --> 00:14:08,750 giving all the reports, answering questions when 342 00:14:08,750 --> 00:14:12,640 we ask questions in class. 343 00:14:12,640 --> 00:14:15,720 Think about how we convince the teams to share out that work 344 00:14:15,720 --> 00:14:17,370 a little bit more evenly. 345 00:14:17,370 --> 00:14:18,870 PHILIP TAN: Well, perhaps, maybe, we 346 00:14:18,870 --> 00:14:23,735 can reduce the extent that we require 347 00:14:23,735 --> 00:14:26,480 these producers to do presentations in class, 348 00:14:26,480 --> 00:14:30,666 and replace that with these more closed-off sessions 349 00:14:30,666 --> 00:14:32,790 where they don't feel like they have to be on stage 350 00:14:32,790 --> 00:14:34,510 and put on the best face for their team. 351 00:14:34,510 --> 00:14:36,080 They can actually just talk about this is the problem 352 00:14:36,080 --> 00:14:37,079 I'm having with my team. 353 00:14:38,597 --> 00:14:40,180 RICHARD EBERHARDT: So one thing I also 354 00:14:40,180 --> 00:14:41,596 wanted to make sure that we talked 355 00:14:41,596 --> 00:14:43,230 about what was it like for you? 356 00:14:43,230 --> 00:14:45,670 How much of your time did we demand? 357 00:14:45,670 --> 00:14:47,640 Was it a lot compared to other projects? 358 00:14:47,640 --> 00:14:48,620 Was it a little bit? 359 00:14:48,620 --> 00:14:50,625 Was it about moderate? 360 00:14:50,625 --> 00:14:53,000 PABLO SUAREZ: I want to start by saying one of the things 361 00:14:53,000 --> 00:14:55,760 that I really appreciated was the time we spent-- 362 00:14:55,760 --> 00:14:58,700 both face-to-face and emailing and so 363 00:14:58,700 --> 00:15:01,550 on-- before the beginning of the course, 364 00:15:01,550 --> 00:15:04,100 to have understanding of what you and your students 365 00:15:04,100 --> 00:15:07,320 can offer, what we need and can offer, 366 00:15:07,320 --> 00:15:09,160 and to manage expectations, as you said. 367 00:15:09,160 --> 00:15:10,990 I think that was very well done. 368 00:15:10,990 --> 00:15:12,590 And it's not always the case. 369 00:15:12,590 --> 00:15:16,620 So with the students and professors and Red Cross, 370 00:15:16,620 --> 00:15:19,480 in general, we [INAUDIBLE], so thank you. 371 00:15:19,480 --> 00:15:21,030 That was really appreciated. 372 00:15:21,030 --> 00:15:27,135 I think that amount of time that I personally had to give 373 00:15:27,135 --> 00:15:30,290 was about as much as I anticipated, about as much 374 00:15:30,290 --> 00:15:33,880 as I think is reasonable to expect for the future. 375 00:15:33,880 --> 00:15:36,770 We do have Jana, in particular, and a few other members 376 00:15:36,770 --> 00:15:39,120 of the team-- of the Red Cross team-- 377 00:15:39,120 --> 00:15:41,900 who were available to give a hand. 378 00:15:41,900 --> 00:15:44,460 And I think we needed to improve because I 379 00:15:44,460 --> 00:15:46,150 did make it very clear that I couldn't 380 00:15:46,150 --> 00:15:49,430 get five emails per day asking questions. 381 00:15:49,430 --> 00:15:53,460 But to have some kind of ability for the right contact when 382 00:15:53,460 --> 00:15:57,220 either engagement levels are too low or mega confusion 383 00:15:57,220 --> 00:16:01,380 levels are too high or sense of purpose is diluted and so on. 384 00:16:01,380 --> 00:16:04,570 So to refine that. 385 00:16:04,570 --> 00:16:09,730 I think it would be better to think of times 386 00:16:09,730 --> 00:16:15,050 when there can be slots of decided interaction 387 00:16:15,050 --> 00:16:18,110 between each team, and someone from the client organization 388 00:16:18,110 --> 00:16:22,050 that doesn't have to be during class. 389 00:16:22,050 --> 00:16:24,494 Because I would always interfere with your plan. 390 00:16:24,494 --> 00:16:25,910 You gave us one opportunity that I 391 00:16:25,910 --> 00:16:28,550 thought was very useful, but a little bit late 392 00:16:28,550 --> 00:16:32,346 in the development process and also very crammed. 393 00:16:32,346 --> 00:16:33,762 By the time we said, OK, these are 394 00:16:33,762 --> 00:16:35,760 the kind of things you can do, there was no time 395 00:16:35,760 --> 00:16:37,135 for them to think and reflect on, 396 00:16:37,135 --> 00:16:40,440 OK, we can do A or B or C. Show that to us, 397 00:16:40,440 --> 00:16:41,960 and then us providing feedback. 398 00:16:41,960 --> 00:16:44,940 So even if it were one more chance via Skype 399 00:16:44,940 --> 00:16:49,960 but scheduled, planned, and if-- it is in the nature of our work 400 00:16:49,960 --> 00:16:52,440 to have to travel, and it is in the nature of our work 401 00:16:52,440 --> 00:16:55,616 that travel timing changes outside of our control. 402 00:16:55,616 --> 00:16:57,740 So where there's a default date, and if things have 403 00:16:57,740 --> 00:16:59,020 to be changed, it is changed. 404 00:16:59,020 --> 00:17:02,420 So to know that there's going to be an instance of feedback that 405 00:17:02,420 --> 00:17:06,830 may be too little, too late, still but to anticipate that. 406 00:17:06,830 --> 00:17:09,079 RICHARD EBERHARDT: Yeah, part of that is the students, 407 00:17:09,079 --> 00:17:11,240 we don't always know the student's schedules. 408 00:17:11,240 --> 00:17:14,130 So it could possibly be planned in the beginning 409 00:17:14,130 --> 00:17:14,910 of the semester. 410 00:17:14,910 --> 00:17:16,500 But planning it before the semester 411 00:17:16,500 --> 00:17:17,910 would be really difficult. Unless we just say it-- 412 00:17:17,910 --> 00:17:19,810 this is the time that you're going to have. 413 00:17:19,810 --> 00:17:21,976 If you don't take advantage of it, you don't get it. 414 00:17:21,976 --> 00:17:24,099 PABLO SUAREZ: Right. 415 00:17:24,099 --> 00:17:26,260 It has to be we propose it's this week. 416 00:17:26,260 --> 00:17:27,540 This is the default time slot. 417 00:17:27,540 --> 00:17:31,036 If not, that time slot, agree on a different time slot. 418 00:17:31,036 --> 00:17:32,410 And then it's up to the students. 419 00:17:32,410 --> 00:17:33,060 RICHARD EBERHARDT: So you're actually 420 00:17:33,060 --> 00:17:35,450 saying you would have talked to them a little bit more. 421 00:17:35,450 --> 00:17:38,230 PABLO SUAREZ: If the problem was lack 422 00:17:38,230 --> 00:17:40,410 of engagement or perception of lack of support. 423 00:17:40,410 --> 00:17:40,720 RICHARD EBERHARDT: So it's really 424 00:17:40,720 --> 00:17:41,770 that planning and scheduling. 425 00:17:41,770 --> 00:17:43,310 So as long as that's in place, then the time 426 00:17:43,310 --> 00:17:44,810 is there, because it's been planned. 427 00:17:44,810 --> 00:17:45,460 OK. 428 00:17:45,460 --> 00:17:46,835 PABLO SUAREZ: And also I think it 429 00:17:46,835 --> 00:17:50,160 would have been nice for them to have access to, say, someone 430 00:17:50,160 --> 00:17:53,030 from the Ghana Team to look at the color, someone 431 00:17:53,030 --> 00:17:57,180 from the Heatwave Team in Buenos Aires-- to at least 432 00:17:57,180 --> 00:17:59,690 to get a flavor, there's another human being there that 433 00:17:59,690 --> 00:18:01,300 eventually can use their game. 434 00:18:01,300 --> 00:18:04,630 And I didn't do enough to help them envision 435 00:18:04,630 --> 00:18:06,730 the user-- either the Red Cross user 436 00:18:06,730 --> 00:18:11,190 or the community level or policy or government user. 437 00:18:11,190 --> 00:18:14,591 PHILIP TAN: I think we need to state that strongly 438 00:18:14,591 --> 00:18:17,410 that it's not just the information that you're 439 00:18:17,410 --> 00:18:18,000 providing. 440 00:18:18,000 --> 00:18:21,240 It's the fact that there is a real person who can really 441 00:18:21,240 --> 00:18:23,830 benefit from the work you're doing is much more 442 00:18:23,830 --> 00:18:25,855 powerful than some sort of abstracted audience. 443 00:18:25,855 --> 00:18:27,230 RICHARD EBERHARDT: So we provided 444 00:18:27,230 --> 00:18:30,990 plenty of documentation, but that's not 445 00:18:30,990 --> 00:18:32,130 the same as a human being. 446 00:18:32,130 --> 00:18:34,472 PHILIP TAN: That's not the emotional drive 447 00:18:34,472 --> 00:18:36,180 that you're going to get-- oh, someone is 448 00:18:36,180 --> 00:18:37,472 counting on me doing good work. 449 00:18:37,472 --> 00:18:39,180 RICHARD EBERHARDT: So maybe it might just 450 00:18:39,180 --> 00:18:42,200 be something for us to talk to the Terrascope folks 451 00:18:42,200 --> 00:18:43,362 to see how they do that. 452 00:18:43,362 --> 00:18:44,820 PHILIP TAN: Well, they specifically 453 00:18:44,820 --> 00:18:50,140 get people who can commit on regular meetings. 454 00:18:50,140 --> 00:18:53,350 And the idea is-- my understanding 455 00:18:53,350 --> 00:18:56,070 is that it's for the domain knowledge primarily. 456 00:18:56,070 --> 00:18:57,890 But you get that side benefit of here 457 00:18:57,890 --> 00:19:00,610 is a real human being who's excited about this problem, 458 00:19:00,610 --> 00:19:06,600 or maybe really concerned about this problem, 459 00:19:06,600 --> 00:19:09,266 and that can influence the motivation of a team. 460 00:19:09,266 --> 00:19:11,390 PABLO SUAREZ: And also, I don't know to what extent 461 00:19:11,390 --> 00:19:13,040 we can keep cramming what students have 462 00:19:13,040 --> 00:19:15,400 to do-- I know they already have to do way too much-- 463 00:19:15,400 --> 00:19:18,790 but to look at how that kind of field 464 00:19:18,790 --> 00:19:22,360 is currently being communicated in the game universe 465 00:19:22,360 --> 00:19:23,210 and outside of that. 466 00:19:23,210 --> 00:19:27,320 So PowerPoints or documents, they're, across the board, 467 00:19:27,320 --> 00:19:29,520 generally boring and unsuccessful-- 468 00:19:29,520 --> 00:19:31,530 even if they have the right content. 469 00:19:31,530 --> 00:19:34,030 So for them to get a sense that if someone really 470 00:19:34,030 --> 00:19:36,770 needs to create awareness on that topic, 471 00:19:36,770 --> 00:19:38,650 and this is the best they have in what 472 00:19:38,650 --> 00:19:42,500 exists in the real world, how about trying to give something 473 00:19:42,500 --> 00:19:46,860 that makes people want to do more through their game? 474 00:19:46,860 --> 00:19:50,740 PHILIP TAN: I think saying, this is how low the bar is, can we 475 00:19:50,740 --> 00:19:53,616 get it slightly better, is less motivating than saying, 476 00:19:53,616 --> 00:19:54,740 what's the best you can do? 477 00:19:58,995 --> 00:20:01,120 RICHARD EBERHARDT: But it's also choosing something 478 00:20:01,120 --> 00:20:02,620 that's going to get used. 479 00:20:02,620 --> 00:20:06,809 Why go in a direction that's-- if this is what those people 480 00:20:06,809 --> 00:20:09,350 are thinking about, if you give them something completely not 481 00:20:09,350 --> 00:20:11,724 like that at all, they're probably not going to know what 482 00:20:11,724 --> 00:20:12,930 to do with it. 483 00:20:12,930 --> 00:20:14,970 PHILIP TAN: But I mean, I can imagine things 484 00:20:14,970 --> 00:20:17,360 like evaluating current methods of teaching 485 00:20:17,360 --> 00:20:19,930 about a certain concept, to be part of that research process 486 00:20:19,930 --> 00:20:22,590 that we've been talking about-- it's understanding the problem. 487 00:20:22,590 --> 00:20:23,590 RICHARD EBERHARDT: Cool. 488 00:20:23,590 --> 00:20:26,760 All right, so I think it's about time to let you go. 489 00:20:26,760 --> 00:20:28,490 But thanks so much for coming in. 490 00:20:28,490 --> 00:20:30,690 PHILIP TAN: We thank you. 491 00:20:30,690 --> 00:20:32,240 Huge gratitude.