Blog Posts

On Monday 11th of April we (Dan, David, Gerhard and me) connected Microsofts Kinect sensor to play around with the amazing 3D signal the gadget provides. The goal was to create a small image carousel, controlled by the sensor. We wanted to try two different approaches: one with Flash and the other one with JavaScript.

Accessing the 3D data with Flash was quite easy since I couldn't wait and already played around with the Kinect before. It also seems that there is already more stuff around for the Flash way (compared to the JavaScript way). We used OpenNI and as3kinect and the results after only few hours were very promising. You can control the carousel by moving your hand left and right. If you want to look at a pictureit in more detail, you can literally grab it and do so. You can then use the familiar multitouch gestures to zoom in and out. When you're finished with that, you can throw (again: literally) the image back into the carousel. Look at the video to get an idea of the demo app:

Some background info: With as3kinect you can access the skeleton and depth map data from OpenNI directly in Flash. You can then track 14 dots of the body in real time. What you don't get yet are gestures (exception: Touch gesture). So we had to do some custom math to recognize things like "swipe", "push", "pull" and so on. There is the middleware NITE which intends to detect these gestures, but it's gesture detection feature is not used in as3kinect (yet).

There is an amazing video demonstrating a browser plugin called DepthJs. But this was all we found that was done with open web techniques. So what we did: David and Dan built a page with HTML5 WebSockets and the jQuery HTML5 Carousel. The client simply connects to the server and listens for messages like "push", " swipeleft" and "swiperight".

For the server side, we wanted to use Processing and thus needed a Java binding of the Kinect signal. We found Processing Simple-OpenNI and added the "push" and "swipe" detectors. To install it, you first need the Kinect drivers, OpenNI and NITE. We were successful with the guide of SensorKinect.

To provide the data over WebSockets, we needed a WebSocket implementation for the Java side. On a first attempt, we found only bloated stuff that brought Tomcat or even JBoss into the game and would have make it pretty complicated to just open a socket connection. After wasting some time trying to "quickly" implement it ourselves (the handshake is really pretty freakish) we finally found the Java-WebSocket library which perfectly suited our needs.

The code we created is available on github.

Here is a shaky demo video:

The hackday was quite a success - we reached our goals and implemented both, a JavaScript and a Flash client. The Kinect sensor really is a great tool. Also it is amazing how many libraries allowing you to do your own stuff are available already. One big question is if this or similar devices will become available for mass market applications. Right now the Kinect is a great gaming interface and an interesting tool to make lab experiments. But what if computers/TVs had built in a 3D sensor out of the box like they have webcams now? The possibilities would be countless, but would we really want to do something like Google Motion? Let's see;) Anyway we are very curious what the Kinect future brings.

Edit: Asus will soon bring a NUI-device for PCs to market. Thx for the hint @sandro