ECCV 2014 Refraction Wiggles for Measuring Fluid Depth and Velocity from Video
Tianfan Xue Michael Rubinstein Neal Wadhwa Anat Levin Fr├ędo Durand William T. Freeman
MIT Computer Science and Artificial Intelligence Laboratory
Microsoft Research
Weizmann Institute of Science

Measuring the velocity and depth of imperceptible candle plumes from standard videos. The heat rising from two burning candles (a, b) cause small distortions of the background due to light rays refracting as they travel from the background to the camera passing through the hot air. Methods such as synthetic Schlieren imaging (c, d) are able to visualize those small disturbances and reveal the heat plume, but are unable to measure its actual motion. We show that, under reasonable conditions, the refraction patterns (observed motions) move coherently with the refracting fluid, allowing to accurately measure the 2D motion of the flow from a monocular video (e), and the depth of the flow from a stereo sequence (f). The full sequence and results are available in the supplementary material.



We present principled algorithms for measuring the velocity and 3D location of refractive fluids, such as hot air or gas, from natural videos with textured backgrounds. Our main observation is that intensity variations related to movements of refractive fluid elements, as observed by one or more video cameras, are consistent over small space-time volumes. We call these intensity variations ``refraction wiggles'', and use them as features for tracking and stereo fusion to recover the fluid motion and depth from video sequences. We give algorithms for 1) measuring the (2D, projected) motion of refractive fluids in monocular videos, and 2) recovering the 3D position of points on the fluid from stereo cameras.

Unlike pixel intensities, wiggles can be extremely subtle and cannot be known with the same level of confidence for all pixels, depending on factors such as background texture and physical properties of the fluid. We thus carefully model uncertainty in our algorithms for robust estimation of fluid motion and depth.

We show results on controlled sequences, synthetic simulations, and natural videos. Different from previous approaches for measuring refractive flow, our methods operate directly on videos captured with ordinary cameras, do not require auxiliary sensors, light sources or designed backgrounds, and can correctly detect the motion and location of refractive fluids even when they are invisible to the naked eye.

@article{xue2014fluidflow,   author = {Tianfan Xue and Michael Rubinstein and Neal Wadhwa and Anat Levin and Fr\'{e}do Durand and William T. Freeman},   title = {Refraction Wiggles for Measuring Fluid Depth and Velocity from Video},   journal = {Proc. of European Conference on Computer Vision (ECCV)},   year = {2014}, }



Paper: pdf

Video Demo: mp4

Supplementary Material: pdf

ECCV Presentation: Video Lecture

Slides: pptx (slides only, ~60 MB), zip (slides + videos, ~500 MB)

Code: zip

Data: zip (Uncompressed gray-scale video)

Result (Video): zip

Result (Mat file): zip


Related Publications

The Aperture Problem for Refractive Motion, T. Xue, H. Mobahi, F. Durand, W. T. Freeman, CVPR 2015


Selected Results
Single View      

result (mp4)

result (mp4)

result (mp4)

result (mp4)


result (mp4)

result (mp4)

result (mp4)



We thank the ECCV reviewers for their comments. Tianfan Xue was supported by Shell International Exploration & Production Inc. and Motion Sensing Wi-Fi Sensor Networks Co. under Grant No. 6925133. Neal Wadhwa was supported by the NSF Graduate Research Fellowship Program under Grant No. 1122374. Part of this work was done when Michael Rubinstein was a student at MIT, supported by the Microsoft Research PhD Fellowship.



Last updated: May 2015