Soap and Milk

We used an industrial infrared camera by Ximea to capture the movements of the users. The system is able to shoot 2k video in 170 frames per second, allowing the algorithm to never lose track of a moving body or its motion. In order to process this data, it uses a custom GPU based optical flow solution to analyze and compare 2 million pixels in under 5 milliseconds.

Within our research we found high frame rate and resolution to be more than just a boost of quality. Basic computer vision for interactive systems is often too slow to measure a user’s movement on a satisfactory level. The captured movement between two frames in time are essential for the algorithms outcome. At 170 fps the solution reconstructs more than the human eye is able to see. The resulting data reveals even the tiniest and fastest motion, allowing the user to feel their dynamics immediately.

The whole system runs on one computer, equipped with a strong processor and a Titan X. The camera image is uploaded to GPU, as soon as a new frame is detected. From this point on, all tasks ranging from image analysis to real-time rendering is performed on GPU only – to avoid bottlenecks of data processing and latency. The motion detection is done in openFrameworks and the rendering passes in vvvv. The communication between the applications is realized via Spout.