Mark Vrablic demonstrates how the Unity game engine connects to the Microsoft Kinect. The guided tutorial that students have completed called “roll-a-ball” is a game normally driven by arrow keys on a standard keyboard. In this video, Mark shows how to modify the game to have the controls be dictated by the position of the user’s hands and other body parts in space.
PROFESSOR: OK. So at this point, we have our roller ball demo. Now, we would like to interface with the Kinect. To do this, we're going to use the simple Kinect Unity package. It's over here. So in the simple Kinect package there is a prefab. You can drag this prefab into your game and it will initiate the Kinect and do all the stuff that's required for that.
So now, when I start my game, the Kinect will turn on. And it's not going to do anything yet because I haven't interfaced it. So, now I want my ball that we created previously to follow my right hand sign. So what we're going to do is go into our player and we're going to add a script. The script that we're going to add is the follow join. You can find it in scripts, no, Kinect stuff. There it is and follow join. So let's drag it on to our player, and then we need to fill out which object we would like to track.
So in this case I'd like to follow my right hand. And you can find that I have a right hand object here. So you can drag this right hand object into the player controller script. So that's the tracked object. And now the ball should follow my right hand.
So you can see it there in the scene. As I move my hand closer to the Kinect, it moves closer to the camera. In many gamines, this isn't what You'd want though. So inside that follow join script there is an to invert each access. Since the z-axis is one goes toward the camera, in this case, we're going to invert the z-axis. So now, when we click play, as I move my hand forward, the ball moves forward within the world space, and back, left, right, up, down, etc. So now, with my right hand, I can control the ball.
All of the pick-ups should still work as they did before and the game can still be won. So I'm getting all the pick-ups with my right hand. Have to go back it up a little. It's just like using a mouse or arrow keys, but in 3D. So as you can see, that still works. Your turn. So now, let's say we want to track something other than our right hand. You can adjust this in the moving object parent. So I have a detect joint script attached to here and this has a track joint option. You can change this to be any joint you like.
So let's say I want to do my left and. I'll select the hand left object, which is somewhere. There it is. And then, for the sake of making things make sense, rename my object to left hand object. So now that this is called left hand object, we want to make sure that the object attached to the player controller, follow join, is indeed the left hand object as is visible right here. So now when I click play, it should follow my left hand not my right hand. If the Kinect can find me. There it is. And so I can move my left hand forward, left, right, up, down, etc. So now, let's say we want to try multiple joints at the same time.
This whole moving objects parent part can be duplicated. So Control D or Command D to duplicate. And we can now create a left had object and a right hand object. So let's go back to being hand-right and rename this.
So I'm going to add another sphere to my scene in order to have another object for my game. So let's create a sphere. And then I'm going it so that my right hand follows this one since my left hand already follows the player. So in Kinect stuff, follow join. Drag it right on to that sphere. OK. So now, I want this to follow my right hand object, so I'm going to drag that in as my track joint. And now that sphere should follow my right hand, while the other follows my left.
So I didn't set the invert option on that one though, so as I move forward, they each go in opposite directions. But otherwise, left and right work as you'd expect, and up and down. So let's say I want to try a different joint. Like, let's make my left hand object be my head. We'll go to that moving object parent , change it to head here. And let's rename this to head.
Now, my right hand controls the sphere that we just added and my head controls the player. So, where is my right hand? There is my right hand. Where is my head? Can we see it? Can I get it in?
It's down here. You can see it up on scene view. I'm struggling to get bearings. Oh, I never set invert axis. That would explain that.
Probably going to--
It was on player, right? No, it was not. It was on the sphere.
Hope I can re-enter myself in world space. There it is. So, as I move my head left and right, up and down, and my hand, they all go properly. Each object's position can be addressed by other objects. And so this is how you can gather your data to use however you like.
