Overview

The Kinect is a depth sensor that produces moderate quality depth and RGB images at a very low cost. The ability to detect depth provides many new ways for robots to sense and analyze their environment. This project integrates the Kinect sensor into Tekkotsu, a framework for programming robots at a high level.

RGB Image Source

The Kinect has a VGA web camera that sends images at 640x480 resolution at 30 frames per second. These images are sent into the Tekkotsu framework in the same way as a webcam, and can be analyzed, segmented, etc. through Tekkotsu. Although the webcam gives raw data as YUV, the OpenNI drivers for the Kinect will automatically convert it to RGB. In order to work with Tekkotsu, it has to be converted back to YUV.

Depth Source

The depth sensor on the Kinect works by projecting several IR dots and then an IR camera can see the positions of these dots. A hardware chip then analyzes dots and generates a 640x480 image at 30 frames per second. Each pixel of this depth map is a short integer, and has values in the range of 0 to 10001. One flaw in the design of the depth sensor is that it can only sense points that are 2 feet or farther away. If an object is closer than 2 feet, it will show up with values of 0. The driver for Tekkotsu converts the depth image from into a YUV image, using the depth as the intensity. This is a great way to visualize the depth map using the Raw Camera viewer. However, this loses information so a the driver also makes a raw depth map available.

Method

The driver consists of the KinectDriver.cc and KinectDriver.h files. The KinectDriver class extends the DeviceDriver class, the main purpose of which is the getImageSources() function that returns a map of sources. These sources are the RGBSource and DepthSource classes. RGBSource and DepthSource are private classes within KinectDriver. They both extend the DataSource class and so they provide some functions that are necessary for a DataSource. These are doFreeze(), a function that tells the data source to stop sending images, doUnfreeze(), a function that tells the data source to start sending images, and advance(), a function that tell the data source to send the next image if available.

OpenNI creates a production tree, similar to the way Tekkotsu has image pipelines. The nodes in the tree that the driver uses are the ImageGenerator and DepthGenerator nodes. These nodes provide new images when they are available. When the doUnfreeze() function is called in the RGBSource or DepthSource, the source registers to its respective generator and listens for updates. When the doFreeze() function is called, they unregister from the generator.

More nodes are possible to be added. There are generators for generating the position of a person, tracking their skeleton, position of their hand, receiving gestures, using the microphone array, and much more. However, most of these features require the NITE library which has proprietary code that can do all this analysis.

When we initialize OpenNI, we need to create a Context and tell it to read the configuration of sensors from a text file. The file that the Tekkotsu Kinect driver uses to read the configuration is project/ms/config/OpenNI-Kinect.xml. This configuration file can be used to setup the sensor configuration, such as the default resolution, pixel type (RGB, YUV, grayscale), or mirroring (vertical reflection).

There is also a HAL configuration file that is needed for the Kinect driver. The driver must be loaded by specifying KinectDriver in the Drivers section, and also by including the Kinect.Camera and Kinect.Depth sources in the Vision section. See kinect.plist for an example.

Since there are two image sources, the RGB image and the Depth image, they are sent into Tekktosu on separate channels. The code included below has the DepthSource's ID set to 0 and the RGBSource's ID set to 1. The raw camera view in ControllerGUI only shows channel 0, but either source can be imported into a sketch.

Results

RGB Image:

Depth:

Installation

Install the prerequisite packages: apt-get install g++ python libusb-1.0.0-dev freeglut3-dev

Follow the installation instructions from http://www.keyboardmods.com/2010/12/howto-kinect-openninite-skeleton.html. The instructions are as follows:

  1. mkdir ~/kinect && cd ~/kinect
  2. git clone https://github.com/OpenNI/OpenNI.git
  3. cd OpenNI/Platform/Linux-x86/Build
  4. make && sudo make install
  5. cd ~/kinect/
  6. git clone https://github.com/boilerbots/Sensor.git
  7. cd Sensor
  8. git checkout kinect
  9. cd Platform/Linux-x86/Build
  10. make && sudo make install

Code